[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250113 08:56:42.100845 10702 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.115.190:46767
I20250113 08:56:42.102375 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:56:42.103451 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:42.120436 10711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:42.121018 10709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:42.122372 10702 server_base.cc:1034] running on GCE node
W20250113 08:56:42.121022 10708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:43.299961 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:56:43.300212 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:56:43.300398 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758603300379 us; error 0 us; skew 500 ppm
I20250113 08:56:43.301203 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:56:43.307832 10702 webserver.cc:458] Webserver started at http://127.10.115.190:35725/ using document root <none> and password file <none>
I20250113 08:56:43.308733 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:43.308954 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:43.309409 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:56:43.315778 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/master-0-root/instance:
uuid: "de95065bb0b64f679d49107727bb2808"
format_stamp: "Formatted at 2025-01-13 08:56:43 on dist-test-slave-lrhs"
I20250113 08:56:43.323414 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250113 08:56:43.328434 10717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:43.329618 10702 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250113 08:56:43.329962 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/master-0-root
uuid: "de95065bb0b64f679d49107727bb2808"
format_stamp: "Formatted at 2025-01-13 08:56:43 on dist-test-slave-lrhs"
I20250113 08:56:43.330305 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:43.378815 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:43.380334 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:56:43.380800 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:43.451191 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.190:46767
I20250113 08:56:43.451288 10768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.190:46767 every 8 connection(s)
I20250113 08:56:43.458129 10769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:43.481791 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808: Bootstrap starting.
I20250113 08:56:43.488233 10769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:43.490126 10769 log.cc:826] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808: Log is configured to *not* fsync() on all Append() calls
I20250113 08:56:43.494726 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808: No bootstrap required, opened a new log
I20250113 08:56:43.517182 10769 raft_consensus.cc:357] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de95065bb0b64f679d49107727bb2808" member_type: VOTER }
I20250113 08:56:43.517864 10769 raft_consensus.cc:383] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:43.518149 10769 raft_consensus.cc:738] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de95065bb0b64f679d49107727bb2808, State: Initialized, Role: FOLLOWER
I20250113 08:56:43.518930 10769 consensus_queue.cc:260] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de95065bb0b64f679d49107727bb2808" member_type: VOTER }
I20250113 08:56:43.519440 10769 raft_consensus.cc:397] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 08:56:43.519726 10769 raft_consensus.cc:491] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 08:56:43.520040 10769 raft_consensus.cc:3054] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:43.524027 10769 raft_consensus.cc:513] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de95065bb0b64f679d49107727bb2808" member_type: VOTER }
I20250113 08:56:43.524763 10769 leader_election.cc:304] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de95065bb0b64f679d49107727bb2808; no voters:
I20250113 08:56:43.526531 10769 leader_election.cc:290] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 08:56:43.526855 10772 raft_consensus.cc:2798] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:43.529009 10772 raft_consensus.cc:695] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 1 LEADER]: Becoming Leader. State: Replica: de95065bb0b64f679d49107727bb2808, State: Running, Role: LEADER
I20250113 08:56:43.529968 10772 consensus_queue.cc:237] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de95065bb0b64f679d49107727bb2808" member_type: VOTER }
I20250113 08:56:43.531028 10769 sys_catalog.cc:564] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 08:56:43.540017 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de95065bb0b64f679d49107727bb2808. Latest consensus state: current_term: 1 leader_uuid: "de95065bb0b64f679d49107727bb2808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de95065bb0b64f679d49107727bb2808" member_type: VOTER } }
I20250113 08:56:43.540740 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [sys.catalog]: This master's current role is: LEADER
I20250113 08:56:43.540136 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de95065bb0b64f679d49107727bb2808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de95065bb0b64f679d49107727bb2808" member_type: VOTER } }
I20250113 08:56:43.541085 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [sys.catalog]: This master's current role is: LEADER
I20250113 08:56:43.544049 10780 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 08:56:43.558075 10780 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 08:56:43.559944 10702 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 08:56:43.575059 10780 catalog_manager.cc:1348] Generated new cluster ID: 163268978c784bfd8274fc1c78d1860f
I20250113 08:56:43.575394 10780 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 08:56:43.634402 10780 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 08:56:43.636546 10780 catalog_manager.cc:1505] Loading token signing keys...
I20250113 08:56:43.655661 10780 catalog_manager.cc:5899] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808: Generated new TSK 0
I20250113 08:56:43.656833 10780 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 08:56:43.691301 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:43.699721 10791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:43.700896 10792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:43.705658 10702 server_base.cc:1034] running on GCE node
W20250113 08:56:43.706916 10794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:43.707893 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:56:43.708135 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:56:43.708330 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758603708308 us; error 0 us; skew 500 ppm
I20250113 08:56:43.708926 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:56:43.713363 10702 webserver.cc:458] Webserver started at http://127.10.115.129:33931/ using document root <none> and password file <none>
I20250113 08:56:43.713879 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:43.714064 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:43.714329 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:56:43.715432 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-0-root/instance:
uuid: "02844150e71e4ea7b8d3c5069b179e6b"
format_stamp: "Formatted at 2025-01-13 08:56:43 on dist-test-slave-lrhs"
I20250113 08:56:43.721109 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250113 08:56:43.725474 10799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:43.726307 10702 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250113 08:56:43.726661 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-0-root
uuid: "02844150e71e4ea7b8d3c5069b179e6b"
format_stamp: "Formatted at 2025-01-13 08:56:43 on dist-test-slave-lrhs"
I20250113 08:56:43.727043 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:43.754719 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:43.756484 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:43.759297 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:43.764739 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:56:43.765066 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:43.765435 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:56:43.765663 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:43.860029 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.129:37529
I20250113 08:56:43.860078 10861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.129:37529 every 8 connection(s)
I20250113 08:56:43.867946 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:43.876914 10866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:43.879803 10867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:43.883153 10702 server_base.cc:1034] running on GCE node
W20250113 08:56:43.884558 10869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:43.885768 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:56:43.886075 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:56:43.886300 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758603886281 us; error 0 us; skew 500 ppm
I20250113 08:56:43.887050 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:56:43.891688 10702 webserver.cc:458] Webserver started at http://127.10.115.130:40019/ using document root <none> and password file <none>
I20250113 08:56:43.892401 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:43.892613 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:43.892881 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:56:43.893007 10862 heartbeater.cc:346] Connected to a master server at 127.10.115.190:46767
I20250113 08:56:43.893530 10862 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:43.894629 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-1-root/instance:
uuid: "91fbac8f141842c6803e02488ae2e454"
format_stamp: "Formatted at 2025-01-13 08:56:43 on dist-test-slave-lrhs"
I20250113 08:56:43.894822 10862 heartbeater.cc:510] Master 127.10.115.190:46767 requested a full tablet report, sending...
I20250113 08:56:43.898313 10734 ts_manager.cc:194] Registered new tserver with Master: 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529)
I20250113 08:56:43.901433 10734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46634
I20250113 08:56:43.902696 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.006s
I20250113 08:56:43.907568 10874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:43.908416 10702 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250113 08:56:43.908699 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-1-root
uuid: "91fbac8f141842c6803e02488ae2e454"
format_stamp: "Formatted at 2025-01-13 08:56:43 on dist-test-slave-lrhs"
I20250113 08:56:43.908979 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:43.924109 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:43.925452 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:43.927768 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:43.931104 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:56:43.931309 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:43.931612 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:56:43.931785 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:44.059639 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.130:43067
I20250113 08:56:44.059760 10936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.130:43067 every 8 connection(s)
I20250113 08:56:44.070181 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:44.079236 10940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:44.082278 10941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:44.085027 10937 heartbeater.cc:346] Connected to a master server at 127.10.115.190:46767
I20250113 08:56:44.085493 10937 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:44.086807 10937 heartbeater.cc:510] Master 127.10.115.190:46767 requested a full tablet report, sending...
I20250113 08:56:44.090996 10734 ts_manager.cc:194] Registered new tserver with Master: 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:44.092432 10702 server_base.cc:1034] running on GCE node
I20250113 08:56:44.093400 10734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46644
W20250113 08:56:44.093842 10943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:44.095062 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:56:44.095315 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:56:44.095512 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758604095496 us; error 0 us; skew 500 ppm
I20250113 08:56:44.096035 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:56:44.099098 10702 webserver.cc:458] Webserver started at http://127.10.115.131:35817/ using document root <none> and password file <none>
I20250113 08:56:44.099581 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:44.099761 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:44.099997 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:56:44.101014 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-2-root/instance:
uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
format_stamp: "Formatted at 2025-01-13 08:56:44 on dist-test-slave-lrhs"
I20250113 08:56:44.106531 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20250113 08:56:44.110471 10948 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:44.112048 10702 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250113 08:56:44.112313 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-2-root
uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
format_stamp: "Formatted at 2025-01-13 08:56:44 on dist-test-slave-lrhs"
I20250113 08:56:44.112555 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736758601939195-10702-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:44.133905 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:44.135463 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:44.139295 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:44.143218 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:56:44.143433 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:44.143705 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:56:44.143857 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:44.282086 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.131:41571
I20250113 08:56:44.282220 11010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.131:41571 every 8 connection(s)
I20250113 08:56:44.298869 11011 heartbeater.cc:346] Connected to a master server at 127.10.115.190:46767
I20250113 08:56:44.299328 11011 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:44.300272 11011 heartbeater.cc:510] Master 127.10.115.190:46767 requested a full tablet report, sending...
I20250113 08:56:44.302400 10734 ts_manager.cc:194] Registered new tserver with Master: 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:44.303061 10702 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01474723s
I20250113 08:56:44.304150 10734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46656
I20250113 08:56:44.306552 10702 test_util.cc:274] Using random seed: -1616144293
I20250113 08:56:44.372674 10734 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46658:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 08:56:44.378650 10734 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 08:56:44.542002 10901 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca4bc430a35487c8c06b493400b352c (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250113 08:56:44.542061 10976 tablet_service.cc:1467] Processing CreateTablet for tablet b4c6dad88d334aeebecc08f18f4ec785 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250113 08:56:44.537829 10902 tablet_service.cc:1467] Processing CreateTablet for tablet b4c6dad88d334aeebecc08f18f4ec785 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250113 08:56:44.546265 10973 tablet_service.cc:1467] Processing CreateTablet for tablet dbbed4a53ed94dd4a54927a541a11929 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250113 08:56:44.549010 10974 tablet_service.cc:1467] Processing CreateTablet for tablet 1edf48023a9b41159bb90511619ef496 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250113 08:56:44.544736 10900 tablet_service.cc:1467] Processing CreateTablet for tablet 1edf48023a9b41159bb90511619ef496 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250113 08:56:44.557402 10971 tablet_service.cc:1467] Processing CreateTablet for tablet 0382b50b8cbb4b26ad00df2a811426d6 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250113 08:56:44.554798 10975 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca4bc430a35487c8c06b493400b352c (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250113 08:56:44.548960 10899 tablet_service.cc:1467] Processing CreateTablet for tablet dbbed4a53ed94dd4a54927a541a11929 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250113 08:56:44.560741 10901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca4bc430a35487c8c06b493400b352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.562799 10900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edf48023a9b41159bb90511619ef496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.561543 10898 tablet_service.cc:1467] Processing CreateTablet for tablet 53b97192388a41b6881eef9331803c23 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250113 08:56:44.563781 10902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c6dad88d334aeebecc08f18f4ec785. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.551333 10972 tablet_service.cc:1467] Processing CreateTablet for tablet 53b97192388a41b6881eef9331803c23 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250113 08:56:44.573278 10899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbbed4a53ed94dd4a54927a541a11929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.572265 10973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbbed4a53ed94dd4a54927a541a11929. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 08:56:44.580487 10782 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:56:44.573662 10970 tablet_service.cc:1467] Processing CreateTablet for tablet d27d8218f7154005b6e5829d2567db6b (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250113 08:56:44.585232 10969 tablet_service.cc:1467] Processing CreateTablet for tablet b8e5506e1cd340f0a7e323883c6e2e61 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250113 08:56:44.586057 10896 tablet_service.cc:1467] Processing CreateTablet for tablet d27d8218f7154005b6e5829d2567db6b (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250113 08:56:44.588995 10896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27d8218f7154005b6e5829d2567db6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.586911 10976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c6dad88d334aeebecc08f18f4ec785. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.587841 10897 tablet_service.cc:1467] Processing CreateTablet for tablet 0382b50b8cbb4b26ad00df2a811426d6 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250113 08:56:44.592862 10895 tablet_service.cc:1467] Processing CreateTablet for tablet b8e5506e1cd340f0a7e323883c6e2e61 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250113 08:56:44.594846 10897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0382b50b8cbb4b26ad00df2a811426d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.596662 10970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27d8218f7154005b6e5829d2567db6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.607306 10969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e5506e1cd340f0a7e323883c6e2e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.610184 10895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e5506e1cd340f0a7e323883c6e2e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.616827 10968 tablet_service.cc:1467] Processing CreateTablet for tablet 000d4553694048038746f484781f7e33 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250113 08:56:44.615329 10894 tablet_service.cc:1467] Processing CreateTablet for tablet 000d4553694048038746f484781f7e33 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250113 08:56:44.620949 10894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 000d4553694048038746f484781f7e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.629846 10898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b97192388a41b6881eef9331803c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.632155 10975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca4bc430a35487c8c06b493400b352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.639228 10972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b97192388a41b6881eef9331803c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.648191 10974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edf48023a9b41159bb90511619ef496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.651454 10971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0382b50b8cbb4b26ad00df2a811426d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.663466 10827 tablet_service.cc:1467] Processing CreateTablet for tablet b4c6dad88d334aeebecc08f18f4ec785 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250113 08:56:44.665374 10968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 000d4553694048038746f484781f7e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.666780 10827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c6dad88d334aeebecc08f18f4ec785. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.670436 10826 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca4bc430a35487c8c06b493400b352c (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250113 08:56:44.679785 10826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca4bc430a35487c8c06b493400b352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.678618 10824 tablet_service.cc:1467] Processing CreateTablet for tablet dbbed4a53ed94dd4a54927a541a11929 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250113 08:56:44.697307 10821 tablet_service.cc:1467] Processing CreateTablet for tablet d27d8218f7154005b6e5829d2567db6b (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250113 08:56:44.700834 10824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbbed4a53ed94dd4a54927a541a11929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.705219 11029 tablet_bootstrap.cc:492] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:44.710078 10819 tablet_service.cc:1467] Processing CreateTablet for tablet 000d4553694048038746f484781f7e33 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250113 08:56:44.688921 10825 tablet_service.cc:1467] Processing CreateTablet for tablet 1edf48023a9b41159bb90511619ef496 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250113 08:56:44.714013 10819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 000d4553694048038746f484781f7e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.715627 10823 tablet_service.cc:1467] Processing CreateTablet for tablet 53b97192388a41b6881eef9331803c23 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250113 08:56:44.722335 10822 tablet_service.cc:1467] Processing CreateTablet for tablet 0382b50b8cbb4b26ad00df2a811426d6 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250113 08:56:44.712968 10820 tablet_service.cc:1467] Processing CreateTablet for tablet b8e5506e1cd340f0a7e323883c6e2e61 (DEFAULT_TABLE table=test-workload [id=8dd4a95d24004dfb9ecd695db3a846c8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250113 08:56:44.728184 10821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27d8218f7154005b6e5829d2567db6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.733717 10822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0382b50b8cbb4b26ad00df2a811426d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.745728 10825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edf48023a9b41159bb90511619ef496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.748764 10823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b97192388a41b6881eef9331803c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.749833 10820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e5506e1cd340f0a7e323883c6e2e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:56:44.826279 11029 tablet_bootstrap.cc:654] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:44.873090 11029 tablet_bootstrap.cc:492] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:44.875011 11030 tablet_bootstrap.cc:492] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:44.876717 11029 ts_tablet_manager.cc:1397] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.180s user 0.013s sys 0.010s
I20250113 08:56:44.886559 11029 raft_consensus.cc:357] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:44.889703 11029 raft_consensus.cc:383] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:44.890185 11029 raft_consensus.cc:738] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:44.892076 11029 consensus_queue.cc:260] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:44.905370 10862 heartbeater.cc:502] Master 127.10.115.190:46767 was elected leader, sending a full tablet report...
I20250113 08:56:44.911216 10937 heartbeater.cc:502] Master 127.10.115.190:46767 was elected leader, sending a full tablet report...
I20250113 08:56:44.919500 11029 ts_tablet_manager.cc:1428] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.042s user 0.004s sys 0.007s
I20250113 08:56:44.941665 11029 tablet_bootstrap.cc:492] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:44.955924 11033 tablet_bootstrap.cc:492] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:44.974548 11030 tablet_bootstrap.cc:654] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:44.983225 11029 tablet_bootstrap.cc:654] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:44.995388 11030 tablet_bootstrap.cc:492] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:44.995908 11030 ts_tablet_manager.cc:1397] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.018s
I20250113 08:56:44.999387 11029 tablet_bootstrap.cc:492] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:44.999308 11030 raft_consensus.cc:357] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:44.999967 11029 ts_tablet_manager.cc:1397] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.014s
I20250113 08:56:45.000180 11030 raft_consensus.cc:383] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.000499 11030 raft_consensus.cc:738] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.001183 11030 consensus_queue.cc:260] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:44.993114 11033 tablet_bootstrap.cc:654] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.002648 11029 raft_consensus.cc:357] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.003432 11029 raft_consensus.cc:383] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.003547 11011 heartbeater.cc:502] Master 127.10.115.190:46767 was elected leader, sending a full tablet report...
I20250113 08:56:45.003741 11029 raft_consensus.cc:738] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.004410 11029 consensus_queue.cc:260] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.008263 11030 ts_tablet_manager.cc:1428] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20250113 08:56:45.009193 11030 tablet_bootstrap.cc:492] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.013878 11029 ts_tablet_manager.cc:1428] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.014s user 0.009s sys 0.001s
I20250113 08:56:45.018719 11029 tablet_bootstrap.cc:492] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:45.019443 11030 tablet_bootstrap.cc:654] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.025601 11029 tablet_bootstrap.cc:654] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.026070 11033 tablet_bootstrap.cc:492] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.026532 11033 ts_tablet_manager.cc:1397] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.007s
I20250113 08:56:45.029544 11033 raft_consensus.cc:357] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.030351 11033 raft_consensus.cc:383] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.030652 11033 raft_consensus.cc:738] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.031293 11033 consensus_queue.cc:260] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.040001 11030 tablet_bootstrap.cc:492] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.040824 11030 ts_tablet_manager.cc:1397] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.000s
W20250113 08:56:45.040777 11012 tablet.cc:2367] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:56:45.043567 11030 raft_consensus.cc:357] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.044221 11030 raft_consensus.cc:383] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.044462 11030 raft_consensus.cc:738] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.045224 11030 consensus_queue.cc:260] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.047966 11030 ts_tablet_manager.cc:1428] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:56:45.048545 11029 tablet_bootstrap.cc:492] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:45.048923 11030 tablet_bootstrap.cc:492] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.049047 11029 ts_tablet_manager.cc:1397] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.015s
I20250113 08:56:45.053612 11040 raft_consensus.cc:491] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.054206 11040 raft_consensus.cc:513] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.053030 11029 raft_consensus.cc:357] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.055328 11029 raft_consensus.cc:383] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.055675 11029 raft_consensus.cc:738] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.057777 11033 ts_tablet_manager.cc:1428] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.031s user 0.009s sys 0.004s
I20250113 08:56:45.058737 11033 tablet_bootstrap.cc:492] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.058383 11029 consensus_queue.cc:260] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.062453 11040 leader_election.cc:290] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.067087 11030 tablet_bootstrap.cc:654] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.076027 11029 ts_tablet_manager.cc:1428] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.026s user 0.005s sys 0.002s
I20250113 08:56:45.079524 11029 tablet_bootstrap.cc:492] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:45.094571 11033 tablet_bootstrap.cc:654] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.100572 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "000d4553694048038746f484781f7e33" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
W20250113 08:56:45.104112 10801 leader_election.cc:343] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.106528 11029 tablet_bootstrap.cc:654] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.116415 11030 tablet_bootstrap.cc:492] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.117358 11030 ts_tablet_manager.cc:1397] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.069s user 0.020s sys 0.004s
I20250113 08:56:45.122749 11033 tablet_bootstrap.cc:492] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.123229 11033 ts_tablet_manager.cc:1397] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.065s user 0.020s sys 0.007s
I20250113 08:56:45.125398 11033 raft_consensus.cc:357] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.126157 11033 raft_consensus.cc:383] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.126508 11033 raft_consensus.cc:738] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.126309 11036 raft_consensus.cc:491] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.127614 11036 raft_consensus.cc:513] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.128221 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "000d4553694048038746f484781f7e33" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" is_pre_election: true
I20250113 08:56:45.128985 10986 raft_consensus.cc:2463] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 0.
I20250113 08:56:45.127955 11033 consensus_queue.cc:260] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.130570 10801 leader_election.cc:304] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters: 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:45.131799 11036 leader_election.cc:290] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.132141 11040 raft_consensus.cc:2798] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.132534 11040 raft_consensus.cc:491] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.132902 11040 raft_consensus.cc:3054] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.126978 11030 raft_consensus.cc:357] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.138547 11030 raft_consensus.cc:383] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.141520 11033 ts_tablet_manager.cc:1428] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20250113 08:56:45.142473 11033 tablet_bootstrap.cc:492] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.142768 11029 tablet_bootstrap.cc:492] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:45.140635 11030 raft_consensus.cc:738] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.146339 11029 ts_tablet_manager.cc:1397] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.004s
I20250113 08:56:45.147486 11030 consensus_queue.cc:260] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.151664 11029 raft_consensus.cc:357] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.152892 11029 raft_consensus.cc:383] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.153503 11029 raft_consensus.cc:738] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.155584 11040 raft_consensus.cc:513] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.157757 11029 consensus_queue.cc:260] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.158255 11040 leader_election.cc:290] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.160225 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "000d4553694048038746f484781f7e33" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454"
W20250113 08:56:45.161469 10801 leader_election.cc:343] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.162712 11030 ts_tablet_manager.cc:1428] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.044s user 0.009s sys 0.000s
I20250113 08:56:45.161538 11029 ts_tablet_manager.cc:1428] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.015s user 0.006s sys 0.004s
I20250113 08:56:45.163729 11030 tablet_bootstrap.cc:492] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.166097 11029 tablet_bootstrap.cc:492] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:45.166541 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "000d4553694048038746f484781f7e33" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:45.167830 10986 raft_consensus.cc:3054] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.175046 11033 tablet_bootstrap.cc:654] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.178850 10986 raft_consensus.cc:2463] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 1.
I20250113 08:56:45.181003 10801 leader_election.cc:304] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters: 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:45.181653 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.182888 11040 raft_consensus.cc:2798] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.183727 11029 tablet_bootstrap.cc:654] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
W20250113 08:56:45.185277 10949 leader_election.cc:343] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.186513 11040 raft_consensus.cc:695] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:45.188395 11040 consensus_queue.cc:237] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.191138 11030 tablet_bootstrap.cc:654] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.208678 11029 tablet_bootstrap.cc:492] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:45.209158 11029 ts_tablet_manager.cc:1397] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.008s
I20250113 08:56:45.210131 11030 tablet_bootstrap.cc:492] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.210613 11030 ts_tablet_manager.cc:1397] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.005s
I20250113 08:56:45.212522 11029 raft_consensus.cc:357] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.213495 11029 raft_consensus.cc:383] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.213791 11029 raft_consensus.cc:738] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.213726 11030 raft_consensus.cc:357] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.214634 11030 raft_consensus.cc:383] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.214948 11030 raft_consensus.cc:738] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.215020 11029 consensus_queue.cc:260] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.215663 11030 consensus_queue.cc:260] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.217793 11029 ts_tablet_manager.cc:1428] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250113 08:56:45.218021 11030 ts_tablet_manager.cc:1428] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250113 08:56:45.219027 11030 tablet_bootstrap.cc:492] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.222178 11029 tablet_bootstrap.cc:492] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:45.224982 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.219369 11033 tablet_bootstrap.cc:492] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.227217 11033 ts_tablet_manager.cc:1397] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.085s user 0.014s sys 0.010s
I20250113 08:56:45.227063 10912 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 0.
I20250113 08:56:45.229800 10950 leader_election.cc:304] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters: 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:56:45.231094 11036 raft_consensus.cc:2798] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.231382 11032 raft_consensus.cc:491] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.231537 11036 raft_consensus.cc:491] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.231925 11036 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.227568 10733 catalog_manager.cc:5526] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 1 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:45.231877 11032 raft_consensus.cc:513] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.235411 11032 leader_election.cc:290] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.259030 11036 raft_consensus.cc:513] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.260411 11029 tablet_bootstrap.cc:654] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.256964 11033 raft_consensus.cc:357] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.261320 11033 raft_consensus.cc:383] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.263372 11036 leader_election.cc:290] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.263414 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:45.263604 11030 tablet_bootstrap.cc:654] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.261899 11033 raft_consensus.cc:738] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
W20250113 08:56:45.264816 10949 leader_election.cc:343] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.266346 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:45.267098 10912 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.269850 11033 consensus_queue.cc:260] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.273787 11033 ts_tablet_manager.cc:1428] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.045s user 0.009s sys 0.008s
I20250113 08:56:45.283190 11032 raft_consensus.cc:491] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.283910 11032 raft_consensus.cc:513] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.292251 11032 leader_election.cc:290] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.289932 10912 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 1.
I20250113 08:56:45.295692 10950 leader_election.cc:304] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters: 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:56:45.296803 11036 raft_consensus.cc:2798] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.298023 11033 tablet_bootstrap.cc:492] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.307988 11030 tablet_bootstrap.cc:492] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.309681 11030 ts_tablet_manager.cc:1397] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.091s user 0.019s sys 0.014s
I20250113 08:56:45.316741 11029 tablet_bootstrap.cc:492] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:45.317449 11029 ts_tablet_manager.cc:1397] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.095s user 0.015s sys 0.013s
I20250113 08:56:45.323243 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b97192388a41b6881eef9331803c23" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" is_pre_election: true
I20250113 08:56:45.323480 11030 raft_consensus.cc:357] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.324317 11030 raft_consensus.cc:383] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.324666 11030 raft_consensus.cc:738] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
W20250113 08:56:45.325498 10876 leader_election.cc:343] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.320734 11029 raft_consensus.cc:357] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.329643 11029 raft_consensus.cc:383] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.326977 11030 consensus_queue.cc:260] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.329994 11029 raft_consensus.cc:738] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.331423 11060 raft_consensus.cc:491] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.327596 11036 raft_consensus.cc:695] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:45.329684 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" is_pre_election: true
I20250113 08:56:45.332142 11060 raft_consensus.cc:513] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
W20250113 08:56:45.335086 10876 leader_election.cc:343] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.335964 11060 leader_election.cc:290] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529)
I20250113 08:56:45.337134 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.338047 10837 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 0.
I20250113 08:56:45.350296 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.350342 11029 consensus_queue.cc:260] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.351680 10949 leader_election.cc:304] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:56:45.352725 10912 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 0.
I20250113 08:56:45.357328 11036 consensus_queue.cc:237] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.359225 11033 tablet_bootstrap.cc:654] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.359812 11062 raft_consensus.cc:2798] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.360260 11062 raft_consensus.cc:491] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.360651 11062 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.364164 11030 ts_tablet_manager.cc:1428] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.053s user 0.010s sys 0.002s
I20250113 08:56:45.366214 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b97192388a41b6881eef9331803c23" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.367033 10837 raft_consensus.cc:2463] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 0.
I20250113 08:56:45.376546 11029 ts_tablet_manager.cc:1428] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.059s user 0.006s sys 0.005s
I20250113 08:56:45.376406 10875 leader_election.cc:304] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters: 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:45.378042 11032 raft_consensus.cc:2798] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.378193 11030 tablet_bootstrap.cc:492] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.378571 11032 raft_consensus.cc:491] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.367695 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.379066 11032 raft_consensus.cc:3054] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.385660 11029 tablet_bootstrap.cc:492] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:45.389510 11032 raft_consensus.cc:513] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.391546 10732 catalog_manager.cc:5526] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:45.401409 11032 leader_election.cc:290] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.402221 11032 raft_consensus.cc:491] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.402709 11032 raft_consensus.cc:513] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.411620 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" is_pre_election: true
W20250113 08:56:45.413179 10876 leader_election.cc:343] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.414976 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b97192388a41b6881eef9331803c23" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
W20250113 08:56:45.418562 10876 leader_election.cc:343] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.418990 11032 leader_election.cc:290] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.419692 11030 tablet_bootstrap.cc:654] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
W20250113 08:56:45.423553 10875 leader_election.cc:343] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.424791 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b97192388a41b6881eef9331803c23" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:45.425480 10836 raft_consensus.cc:3054] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.424031 10875 leader_election.cc:304] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91fbac8f141842c6803e02488ae2e454; no voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:45.435163 11033 tablet_bootstrap.cc:492] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.435652 11033 ts_tablet_manager.cc:1397] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.142s user 0.017s sys 0.019s
I20250113 08:56:45.436177 11029 tablet_bootstrap.cc:654] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.436512 11066 raft_consensus.cc:2743] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:56:45.437237 11030 tablet_bootstrap.cc:492] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.437693 11030 ts_tablet_manager.cc:1397] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.013s
I20250113 08:56:45.433640 11062 raft_consensus.cc:513] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.441613 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.441717 11030 raft_consensus.cc:357] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.442458 11030 raft_consensus.cc:383] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.442745 11030 raft_consensus.cc:738] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
W20250113 08:56:45.442938 10875 leader_election.cc:343] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.443531 10875 leader_election.cc:304] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91fbac8f141842c6803e02488ae2e454; no voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:45.443444 11030 consensus_queue.cc:260] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.444432 11066 raft_consensus.cc:2743] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:56:45.443580 11033 raft_consensus.cc:357] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.445362 11033 raft_consensus.cc:383] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.445773 11033 raft_consensus.cc:738] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.446211 11030 ts_tablet_manager.cc:1428] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250113 08:56:45.447705 11030 tablet_bootstrap.cc:492] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.448000 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:45.448588 10837 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.451542 11033 consensus_queue.cc:260] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.458221 10837 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 1.
I20250113 08:56:45.459986 11033 ts_tablet_manager.cc:1428] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.024s user 0.004s sys 0.003s
I20250113 08:56:45.460281 11030 tablet_bootstrap.cc:654] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.464598 10949 leader_election.cc:304] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:56:45.462647 11029 tablet_bootstrap.cc:492] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:45.465384 11036 raft_consensus.cc:2798] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.465385 11062 leader_election.cc:290] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529)
I20250113 08:56:45.465868 11036 raft_consensus.cc:695] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:45.467002 11033 tablet_bootstrap.cc:492] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.466727 11036 consensus_queue.cc:237] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:45.469604 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:45.470187 10912 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.471542 11029 ts_tablet_manager.cc:1397] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.027s
I20250113 08:56:45.475154 11029 raft_consensus.cc:357] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.477061 11033 tablet_bootstrap.cc:654] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.463658 10836 raft_consensus.cc:2463] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 1.
I20250113 08:56:45.477941 10912 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 1.
I20250113 08:56:45.481001 11040 raft_consensus.cc:491] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.481526 11040 raft_consensus.cc:513] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.482980 10875 leader_election.cc:304] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters: 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:45.483347 11040 leader_election.cc:290] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.484490 11066 raft_consensus.cc:2798] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.485160 11029 raft_consensus.cc:383] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.486052 11029 raft_consensus.cc:738] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.484954 11066 raft_consensus.cc:695] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 1 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:56:45.485435 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.486222 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" is_pre_election: true
W20250113 08:56:45.488921 10801 leader_election.cc:343] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.488098 11029 consensus_queue.cc:260] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.489060 10912 raft_consensus.cc:2463] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 0.
I20250113 08:56:45.491284 11030 tablet_bootstrap.cc:492] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.491747 11030 ts_tablet_manager.cc:1397] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.003s
I20250113 08:56:45.488332 11066 consensus_queue.cc:237] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.492295 11029 ts_tablet_manager.cc:1428] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250113 08:56:45.492199 10801 leader_election.cc:304] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters: 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:45.497553 11040 raft_consensus.cc:2798] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.498072 11040 raft_consensus.cc:491] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.498708 11040 raft_consensus.cc:3054] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.498287 11029 tablet_bootstrap.cc:492] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:45.508239 11040 raft_consensus.cc:513] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.511929 11040 leader_election.cc:290] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.513679 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:45.512979 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:45.514828 10912 raft_consensus.cc:3054] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.512859 10732 catalog_manager.cc:5526] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 1 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:45.516845 11033 tablet_bootstrap.cc:492] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.519172 11030 raft_consensus.cc:357] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.520004 11030 raft_consensus.cc:383] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.520341 11030 raft_consensus.cc:738] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.517324 11033 ts_tablet_manager.cc:1397] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.019s
W20250113 08:56:45.524034 10801 leader_election.cc:343] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.521739 11030 consensus_queue.cc:260] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.521795 10731 catalog_manager.cc:5526] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:45.530108 10912 raft_consensus.cc:2463] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 1.
I20250113 08:56:45.529964 11033 raft_consensus.cc:357] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.530766 11033 raft_consensus.cc:383] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.531091 11033 raft_consensus.cc:738] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.531191 10801 leader_election.cc:304] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters: 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:45.531992 11033 consensus_queue.cc:260] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.532573 11040 raft_consensus.cc:2798] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.534634 11033 ts_tablet_manager.cc:1428] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
I20250113 08:56:45.537946 11040 raft_consensus.cc:695] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:45.538533 11033 tablet_bootstrap.cc:492] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.539072 11040 consensus_queue.cc:237] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.543964 11029 tablet_bootstrap.cc:654] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.548069 11033 tablet_bootstrap.cc:654] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.551246 11030 ts_tablet_manager.cc:1428] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.059s user 0.009s sys 0.002s
I20250113 08:56:45.552232 11030 tablet_bootstrap.cc:492] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.559989 11030 tablet_bootstrap.cc:654] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.562868 10733 catalog_manager.cc:5526] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 1 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:45.579736 11029 tablet_bootstrap.cc:492] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:45.580272 11029 ts_tablet_manager.cc:1397] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.010s
I20250113 08:56:45.583145 11030 tablet_bootstrap.cc:492] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.582885 11029 raft_consensus.cc:357] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.583550 11029 raft_consensus.cc:383] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.583693 11030 ts_tablet_manager.cc:1397] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.000s
I20250113 08:56:45.583822 11029 raft_consensus.cc:738] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
W20250113 08:56:45.584618 10782 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:56:45.584486 11029 consensus_queue.cc:260] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.586630 11029 ts_tablet_manager.cc:1428] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250113 08:56:45.586951 11030 raft_consensus.cc:357] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.587657 11030 raft_consensus.cc:383] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.587935 11030 raft_consensus.cc:738] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.588918 11033 tablet_bootstrap.cc:492] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.589390 11033 ts_tablet_manager.cc:1397] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.051s user 0.020s sys 0.021s
I20250113 08:56:45.589550 11030 consensus_queue.cc:260] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.591871 11030 ts_tablet_manager.cc:1428] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250113 08:56:45.601156 11030 tablet_bootstrap.cc:492] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64: Bootstrap starting.
I20250113 08:56:45.602249 11036 raft_consensus.cc:491] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.592252 11033 raft_consensus.cc:357] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.602751 11036 raft_consensus.cc:513] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.603307 11033 raft_consensus.cc:383] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.603631 11033 raft_consensus.cc:738] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.604317 11033 consensus_queue.cc:260] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.605683 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.604979 11036 leader_election.cc:290] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.606885 11033 ts_tablet_manager.cc:1428] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
W20250113 08:56:45.607163 10949 leader_election.cc:343] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.592662 11029 tablet_bootstrap.cc:492] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: Bootstrap starting.
I20250113 08:56:45.607561 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
W20250113 08:56:45.609606 10950 leader_election.cc:343] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:45.610312 10950 leader_election.cc:304] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64; no voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:45.612030 11036 raft_consensus.cc:2743] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:56:45.612867 11033 tablet_bootstrap.cc:492] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.615612 11029 tablet_bootstrap.cc:654] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.619185 11030 tablet_bootstrap.cc:654] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.629922 11033 tablet_bootstrap.cc:654] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.630179 11029 tablet_bootstrap.cc:492] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: No bootstrap required, opened a new log
I20250113 08:56:45.630657 11029 ts_tablet_manager.cc:1397] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.003s
I20250113 08:56:45.633224 11029 raft_consensus.cc:357] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.633946 11029 raft_consensus.cc:383] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.634199 11029 raft_consensus.cc:738] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.634797 11029 consensus_queue.cc:260] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.640810 11029 ts_tablet_manager.cc:1428] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20250113 08:56:45.642817 11033 tablet_bootstrap.cc:492] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.643525 11033 ts_tablet_manager.cc:1397] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.012s
I20250113 08:56:45.646903 11033 raft_consensus.cc:357] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.647809 11033 raft_consensus.cc:383] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.648272 11033 raft_consensus.cc:738] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.648967 11033 consensus_queue.cc:260] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.650936 11030 tablet_bootstrap.cc:492] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64: No bootstrap required, opened a new log
I20250113 08:56:45.651369 11030 ts_tablet_manager.cc:1397] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.008s
I20250113 08:56:45.652366 11033 ts_tablet_manager.cc:1428] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20250113 08:56:45.653666 11033 tablet_bootstrap.cc:492] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.654824 11030 raft_consensus.cc:357] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.655500 11030 raft_consensus.cc:383] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.655732 11030 raft_consensus.cc:738] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.657159 11030 consensus_queue.cc:260] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.660336 11030 ts_tablet_manager.cc:1428] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20250113 08:56:45.662276 11033 tablet_bootstrap.cc:654] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.668078 11033 tablet_bootstrap.cc:492] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.668537 11033 ts_tablet_manager.cc:1397] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20250113 08:56:45.671041 11033 raft_consensus.cc:357] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.671625 11033 raft_consensus.cc:383] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.671880 11033 raft_consensus.cc:738] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.672478 11033 consensus_queue.cc:260] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.674799 11033 ts_tablet_manager.cc:1428] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20250113 08:56:45.675843 11033 tablet_bootstrap.cc:492] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b: Bootstrap starting.
I20250113 08:56:45.682256 11033 tablet_bootstrap.cc:654] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b: Neither blocks nor log segments found. Creating new log.
I20250113 08:56:45.682260 10912 raft_consensus.cc:3054] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.692719 11040 consensus_queue.cc:1035] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:45.694407 11033 tablet_bootstrap.cc:492] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b: No bootstrap required, opened a new log
I20250113 08:56:45.694854 11033 ts_tablet_manager.cc:1397] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20250113 08:56:45.697983 11033 raft_consensus.cc:357] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.698787 11033 raft_consensus.cc:383] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:56:45.699082 11033 raft_consensus.cc:738] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Initialized, Role: FOLLOWER
I20250113 08:56:45.699601 11036 raft_consensus.cc:491] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.699831 11033 consensus_queue.cc:260] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.700053 11036 raft_consensus.cc:513] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.701967 11036 leader_election.cc:290] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.702415 11036 raft_consensus.cc:491] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.703119 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.710621 11033 ts_tablet_manager.cc:1428] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20250113 08:56:45.710778 10911 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 0.
I20250113 08:56:45.712388 10950 leader_election.cc:304] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:45.713451 11062 raft_consensus.cc:2798] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.713871 11062 raft_consensus.cc:491] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.714265 11062 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.704746 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.718505 10836 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 0.
I20250113 08:56:45.709975 11036 raft_consensus.cc:513] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.722591 11063 raft_consensus.cc:491] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.725986 11060 raft_consensus.cc:491] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.723773 11063 raft_consensus.cc:513] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.726464 11060 raft_consensus.cc:513] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.726554 11062 raft_consensus.cc:513] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.727962 11063 leader_election.cc:290] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.728513 11060 leader_election.cc:290] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.731812 11036 leader_election.cc:290] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.733883 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.733904 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b97192388a41b6881eef9331803c23" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.734655 10835 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 0.
I20250113 08:56:45.734848 11062 leader_election.cc:290] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.735707 10836 raft_consensus.cc:2388] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in current term 1: Already voted for candidate 91fbac8f141842c6803e02488ae2e454 in this term.
I20250113 08:56:45.739060 10910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.738488 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b97192388a41b6881eef9331803c23" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.739845 10910 raft_consensus.cc:2388] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in current term 1: Already voted for candidate 02844150e71e4ea7b8d3c5069b179e6b in this term.
I20250113 08:56:45.737140 10949 leader_election.cc:304] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:56:45.740813 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:45.740911 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.741472 11062 raft_consensus.cc:2798] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.733820 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:45.741619 10911 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 0.
I20250113 08:56:45.741983 11062 raft_consensus.cc:491] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.742370 11062 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.743041 10950 leader_election.cc:304] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64; no voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:45.743813 11036 raft_consensus.cc:3054] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.741532 10909 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.746579 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:45.747231 10836 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.749437 11062 raft_consensus.cc:513] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.751459 11036 raft_consensus.cc:2743] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:56:45.752588 11040 consensus_queue.cc:1035] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:45.754433 10835 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.754537 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:45.755174 10837 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.759135 11062 leader_election.cc:290] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:45.762655 10909 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 1.
I20250113 08:56:45.757159 10950 leader_election.cc:304] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64; no voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:45.766657 10836 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 1.
I20250113 08:56:45.768111 10949 leader_election.cc:304] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:56:45.769161 11060 raft_consensus.cc:3054] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.770305 11036 raft_consensus.cc:2798] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.770839 11036 raft_consensus.cc:695] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:45.772892 11036 consensus_queue.cc:237] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.781404 11062 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:45.781885 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:45.782480 10909 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.784351 11060 raft_consensus.cc:2743] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:56:45.788117 10837 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 1.
I20250113 08:56:45.794018 10909 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 1.
I20250113 08:56:45.794575 11060 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:56:45.796263 10950 leader_election.cc:304] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:45.797767 11036 raft_consensus.cc:2798] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.803606 10733 catalog_manager.cc:5526] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:45.798219 11036 raft_consensus.cc:695] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:45.805780 11036 consensus_queue.cc:237] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.830103 10733 catalog_manager.cc:5526] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:45.923573 11066 consensus_queue.cc:1035] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:45.924681 11086 raft_consensus.cc:491] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:45.925127 11086 raft_consensus.cc:513] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.927347 11086 leader_election.cc:290] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.928328 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" is_pre_election: true
I20250113 08:56:45.929104 10985 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 0.
I20250113 08:56:45.930408 10801 leader_election.cc:304] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:56:45.931042 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454" is_pre_election: true
I20250113 08:56:45.931555 11086 raft_consensus.cc:2798] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:45.931720 10911 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 0.
I20250113 08:56:45.931983 11086 raft_consensus.cc:491] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:45.932371 11086 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.934373 11032 consensus_queue.cc:1035] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:45.940608 11086 raft_consensus.cc:513] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.943578 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:45.944319 10911 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.954615 11086 leader_election.cc:290] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:45.955067 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:45.955821 10986 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:45.955283 10911 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 1.
I20250113 08:56:45.959553 10801 leader_election.cc:304] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:45.960592 11086 raft_consensus.cc:2798] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:45.961206 11086 raft_consensus.cc:695] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:45.962663 11086 consensus_queue.cc:237] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:45.970124 10986 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 1.
I20250113 08:56:45.973783 10733 catalog_manager.cc:5526] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 1 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:45.995368 11036 consensus_queue.cc:1035] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.005170 11062 consensus_queue.cc:1035] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:56:46.021944 11066 raft_consensus.cc:491] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:46.022432 11066 raft_consensus.cc:513] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.024255 11066 leader_election.cc:290] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.024896 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b" is_pre_election: true
I20250113 08:56:46.025310 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" is_pre_election: true
I20250113 08:56:46.025960 10986 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 0.
I20250113 08:56:46.025476 10835 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 0.
I20250113 08:56:46.026998 10876 leader_election.cc:304] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.027691 11066 raft_consensus.cc:2798] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:56:46.028025 11066 raft_consensus.cc:491] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:46.028337 11066 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:46.034488 11066 raft_consensus.cc:513] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.036581 11066 leader_election.cc:290] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.036880 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:46.037084 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:46.037714 10986 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:46.037683 10835 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:56:46.043635 10835 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 1.
I20250113 08:56:46.043635 10986 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 1.
I20250113 08:56:46.044852 10875 leader_election.cc:304] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.045559 11066 raft_consensus.cc:2798] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:56:46.046025 11066 raft_consensus.cc:695] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:56:46.047230 11066 consensus_queue.cc:237] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.049643 11086 consensus_queue.cc:1035] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.061483 10733 catalog_manager.cc:5526] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 1 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:46.079438 11086 consensus_queue.cc:1035] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.247854 11036 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.268687 11062 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:56:46.281487 10986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
mode: GRACEFUL
new_leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:35196
I20250113 08:56:46.282153 10986 raft_consensus.cc:604] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Received request to transfer leadership to TS 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:56:46.288095 10986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
mode: GRACEFUL
new_leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:35196
I20250113 08:56:46.288580 10986 raft_consensus.cc:604] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Received request to transfer leadership to TS 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:56:46.292776 10986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
mode: GRACEFUL
new_leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:35196
I20250113 08:56:46.293352 10986 raft_consensus.cc:604] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Received request to transfer leadership to TS 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:56:46.297924 10986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
mode: GRACEFUL
new_leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:35196
I20250113 08:56:46.298559 10986 raft_consensus.cc:604] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Received request to transfer leadership to TS 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:56:46.316457 10835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
mode: GRACEFUL
new_leader_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:34712
I20250113 08:56:46.316989 10835 raft_consensus.cc:604] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Received request to transfer leadership to TS 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:46.321492 10835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
mode: GRACEFUL
new_leader_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:34712
I20250113 08:56:46.321941 10835 raft_consensus.cc:604] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Received request to transfer leadership to TS 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:46.326560 10837 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "000d4553694048038746f484781f7e33"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
mode: GRACEFUL
new_leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:34712
I20250113 08:56:46.327113 10837 raft_consensus.cc:604] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Received request to transfer leadership to TS 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:46.327179 11062 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.334177 11060 raft_consensus.cc:988] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64: : Instructing follower 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.334638 11063 raft_consensus.cc:1076] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Signalling peer 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.337986 10836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:34690
I20250113 08:56:46.338605 10836 raft_consensus.cc:491] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:46.338918 10836 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.342592 11063 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.345285 10836 raft_consensus.cc:513] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.347679 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:46.348093 10836 leader_election.cc:290] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.351359 10837 raft_consensus.cc:1235] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Rejecting Update request from peer 2e6996c7a04c4c3bb45b5defaa8cab64 for earlier term 1. Current term is 2. Ops: []
I20250113 08:56:46.351389 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:46.352322 10986 raft_consensus.cc:3049] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:46.352799 10986 raft_consensus.cc:738] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:46.353857 10986 consensus_queue.cc:260] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.355432 10986 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.362568 10986 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 2.
I20250113 08:56:46.363539 10801 leader_election.cc:304] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters: 91fbac8f141842c6803e02488ae2e454
I20250113 08:56:46.364109 11086 raft_consensus.cc:2798] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:46.364544 11086 raft_consensus.cc:695] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:46.365159 11086 consensus_queue.cc:237] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.374892 10733 catalog_manager.cc:5526] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 1 to 2, leader changed from 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131) to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 2 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:46.394332 11086 consensus_queue.cc:1035] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:56:46.402617 11050 raft_consensus.cc:988] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b: : Instructing follower 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:56:46.403090 11050 raft_consensus.cc:1076] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Signalling peer 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:56:46.405174 10909 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:50812
I20250113 08:56:46.405794 10909 raft_consensus.cc:491] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:46.406246 10909 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.410939 11040 consensus_queue.cc:1035] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.413808 10909 raft_consensus.cc:513] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.414902 10911 raft_consensus.cc:1235] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Rejecting Update request from peer 02844150e71e4ea7b8d3c5069b179e6b for earlier term 1. Current term is 2. Ops: []
I20250113 08:56:46.417063 11040 consensus_queue.cc:1046] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:56:46.417907 11040 raft_consensus.cc:3049] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:46.418215 11040 raft_consensus.cc:738] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:46.418975 11040 consensus_queue.cc:260] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.419953 11040 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.421622 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
W20250113 08:56:46.422267 11050 consensus_queue.cc:1159] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20250113 08:56:46.422549 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:46.423218 10986 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.423629 10909 leader_election.cc:290] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 2 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.433329 10986 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 2.
I20250113 08:56:46.436192 10837 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 2.
I20250113 08:56:46.439375 10875 leader_election.cc:304] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.440346 11032 raft_consensus.cc:2798] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:46.440829 11032 raft_consensus.cc:695] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:56:46.441730 11032 consensus_queue.cc:237] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.449175 10732 catalog_manager.cc:5526] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 1 to 2, leader changed from 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129) to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 2 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:46.453514 11050 raft_consensus.cc:988] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b: : Instructing follower 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:56:46.454052 11040 raft_consensus.cc:1076] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Signalling peer 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:56:46.455585 10909 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:50812
I20250113 08:56:46.456202 10909 raft_consensus.cc:491] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:46.456565 10909 raft_consensus.cc:3054] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.462236 10909 raft_consensus.cc:513] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.464246 10909 leader_election.cc:290] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 2 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.464823 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:46.464902 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e5506e1cd340f0a7e323883c6e2e61" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:46.465767 10986 raft_consensus.cc:3054] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.465711 10837 raft_consensus.cc:3049] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:46.466409 10837 raft_consensus.cc:738] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:46.467278 10837 consensus_queue.cc:260] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.468705 10837 raft_consensus.cc:3054] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.470661 10986 raft_consensus.cc:2463] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 2.
I20250113 08:56:46.471580 10876 leader_election.cc:304] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.472395 11032 raft_consensus.cc:2798] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:46.474735 11032 raft_consensus.cc:695] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 2 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:56:46.475170 10837 raft_consensus.cc:2463] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 2.
I20250113 08:56:46.475567 11032 consensus_queue.cc:237] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.479084 11032 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:56:46.482705 10732 catalog_manager.cc:5526] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 1 to 2, leader changed from 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129) to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 2 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:46.493620 11066 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:56:46.551909 11063 raft_consensus.cc:988] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64: : Instructing follower 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.552376 11062 raft_consensus.cc:1076] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Signalling peer 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.553725 10836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:34690
I20250113 08:56:46.554176 10836 raft_consensus.cc:491] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:46.554440 10836 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.558593 10836 raft_consensus.cc:513] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:46.560093 10836 leader_election.cc:290] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.560910 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:46.561189 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:46.561532 10909 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.561743 10986 raft_consensus.cc:3049] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:46.562057 10986 raft_consensus.cc:738] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:46.562717 10986 consensus_queue.cc:260] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:46.563946 10986 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.568449 10909 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 2.
I20250113 08:56:46.568913 10986 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 2.
I20250113 08:56:46.569761 10801 leader_election.cc:304] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.570696 11040 raft_consensus.cc:2798] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:46.571118 11040 raft_consensus.cc:695] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:46.571890 11040 consensus_queue.cc:237] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:56:46.578219 10732 catalog_manager.cc:5526] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 1 to 2, leader changed from 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131) to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 2 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:46.680179 11050 raft_consensus.cc:988] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b: : Instructing follower 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:56:46.680613 11040 raft_consensus.cc:1076] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Signalling peer 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:56:46.681980 10986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "000d4553694048038746f484781f7e33"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:35190
I20250113 08:56:46.682439 10986 raft_consensus.cc:491] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:46.682716 10986 raft_consensus.cc:3054] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.687067 10986 raft_consensus.cc:513] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.688553 10986 leader_election.cc:290] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 2 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:46.689448 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "000d4553694048038746f484781f7e33" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:46.689520 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "000d4553694048038746f484781f7e33" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:46.690232 10909 raft_consensus.cc:3054] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.690227 10836 raft_consensus.cc:3049] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:46.690747 10836 raft_consensus.cc:738] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:46.691540 10836 consensus_queue.cc:260] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.692718 10836 raft_consensus.cc:3054] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.696625 10909 raft_consensus.cc:2463] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 2.
I20250113 08:56:46.698010 10950 leader_election.cc:304] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.698778 11062 raft_consensus.cc:2798] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:46.698989 10836 raft_consensus.cc:2463] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 2.
I20250113 08:56:46.699295 11062 raft_consensus.cc:695] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:46.700192 11062 consensus_queue.cc:237] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.706038 10732 catalog_manager.cc:5526] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 1 to 2, leader changed from 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129) to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:46.738409 11060 raft_consensus.cc:988] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64: : Instructing follower 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.738880 11062 raft_consensus.cc:1076] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Signalling peer 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.740494 10836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:34690
I20250113 08:56:46.741202 10836 raft_consensus.cc:491] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:46.741616 10836 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.748442 10836 raft_consensus.cc:513] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.750607 10836 leader_election.cc:290] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.751626 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:46.751708 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:46.752314 10911 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.752462 10986 raft_consensus.cc:3049] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:46.752990 10986 raft_consensus.cc:738] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:46.753921 10986 consensus_queue.cc:260] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.755616 10986 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.759241 10911 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 2.
I20250113 08:56:46.760393 10801 leader_election.cc:304] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.761317 11040 raft_consensus.cc:2798] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:46.761794 11040 raft_consensus.cc:695] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:46.762140 10986 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 2.
I20250113 08:56:46.763072 11040 consensus_queue.cc:237] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.771019 10732 catalog_manager.cc:5526] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 1 to 2, leader changed from 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131) to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 2 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:46.818074 10911 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.820030 11060 raft_consensus.cc:988] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: : Instructing follower 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.820529 11062 raft_consensus.cc:1076] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Signalling peer 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:56:46.822360 10836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:34690
I20250113 08:56:46.822952 10836 raft_consensus.cc:491] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:46.823328 10836 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.823813 10911 raft_consensus.cc:1270] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:46.825284 11040 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:56:46.829813 10836 raft_consensus.cc:513] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.833729 10986 raft_consensus.cc:1270] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:46.834098 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:46.834676 10911 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.835491 11040 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:46.846915 10911 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 2.
I20250113 08:56:46.847254 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:56:46.848037 10986 raft_consensus.cc:3049] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:46.848088 10909 raft_consensus.cc:1235] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Rejecting Update request from peer 2e6996c7a04c4c3bb45b5defaa8cab64 for earlier term 1. Current term is 2. Ops: []
I20250113 08:56:46.848482 10986 raft_consensus.cc:738] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:46.849285 10986 consensus_queue.cc:260] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.850044 10986 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:46.854053 10801 leader_election.cc:304] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:46.842034 10836 leader_election.cc:290] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 2 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:56:46.855050 11114 raft_consensus.cc:2798] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:46.855610 11114 raft_consensus.cc:695] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:46.856462 10986 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 2.
I20250113 08:56:46.857089 11114 consensus_queue.cc:237] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:46.858294 10836 raft_consensus.cc:1270] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:46.859549 11110 consensus_queue.cc:1035] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:46.864499 10731 catalog_manager.cc:5526] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 1 to 2, leader changed from 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131) to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 2 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:46.866887 10986 raft_consensus.cc:1270] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:46.868013 11032 consensus_queue.cc:1035] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:46.970391 10836 raft_consensus.cc:1270] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:46.971556 11032 consensus_queue.cc:1035] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:46.980504 10986 raft_consensus.cc:1270] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:46.982105 11032 consensus_queue.cc:1035] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:56:47.024619 10985 raft_consensus.cc:1270] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.025830 11114 consensus_queue.cc:1035] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:47.034080 10909 raft_consensus.cc:1270] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.035367 11114 consensus_queue.cc:1035] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:47.217074 10836 raft_consensus.cc:1270] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.218214 11062 consensus_queue.cc:1035] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:47.226017 10909 raft_consensus.cc:1270] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.227753 11062 consensus_queue.cc:1035] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:47.265174 10985 raft_consensus.cc:1270] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.266487 11114 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:47.272656 10909 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.274446 10911 raft_consensus.cc:1270] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.276166 11114 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:47.276688 11050 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:47.292349 10985 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:47.294384 11122 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:56:49.353870 10702 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20250113 08:56:49.354625 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:49.374807 10911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
mode: GRACEFUL
new_leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:50820
I20250113 08:56:49.375312 10911 raft_consensus.cc:604] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 LEADER]: Received request to transfer leadership to TS 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:49.384608 10836 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
mode: GRACEFUL
new_leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:34720
I20250113 08:56:49.385105 10836 raft_consensus.cc:604] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Received request to transfer leadership to TS 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:56:49.386116 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250113 08:56:49.386564 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:49.416507 11129 raft_consensus.cc:988] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: : Instructing follower 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:56:49.417032 11129 raft_consensus.cc:1076] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 LEADER]: Signalling peer 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:56:49.418609 10985 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:35192
I20250113 08:56:49.419155 10985 raft_consensus.cc:491] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:49.419497 10985 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:49.424120 10985 raft_consensus.cc:513] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:49.425603 10985 leader_election.cc:290] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 2 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:49.426446 10836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:49.426833 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:49.427403 10911 raft_consensus.cc:3049] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:56:49.427723 10911 raft_consensus.cc:738] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:56:49.428637 10911 consensus_queue.cc:260] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:49.429677 10911 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:49.433976 10911 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 2.
I20250113 08:56:49.434886 10950 leader_election.cc:304] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters: 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:56:49.435528 11130 raft_consensus.cc:2798] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:56:49.436549 11130 raft_consensus.cc:695] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:49.437774 11130 consensus_queue.cc:237] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:49.445809 10731 catalog_manager.cc:5526] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 1 to 2, leader changed from 91fbac8f141842c6803e02488ae2e454 (127.10.115.130) to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:49.497859 11050 raft_consensus.cc:988] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b: : Instructing follower 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:56:49.498272 11050 raft_consensus.cc:1076] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Signalling peer 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:56:49.499464 10985 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:35190
I20250113 08:56:49.499927 10985 raft_consensus.cc:491] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:49.500205 10985 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:56:49.504282 10985 raft_consensus.cc:513] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:49.505731 10985 leader_election.cc:290] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 3 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:56:49.506597 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:56:49.506671 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:56:49.507061 10837 raft_consensus.cc:3049] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:56:49.507153 10911 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:56:49.507321 10837 raft_consensus.cc:738] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:56:49.507972 10837 consensus_queue.cc:260] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:49.508852 10837 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:56:49.512020 10911 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 3.
I20250113 08:56:49.512897 10950 leader_election.cc:304] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:56:49.513262 10837 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 3.
I20250113 08:56:49.513746 11130 raft_consensus.cc:2798] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:56:49.514137 11130 raft_consensus.cc:695] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:56:49.514858 11130 consensus_queue.cc:237] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:56:49.521200 10731 catalog_manager.cc:5526] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 2 to 3, leader changed from 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129) to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 3 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:50.610492 10837 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:56:50.610680 10836 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:56:50.614389 11148 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:56:50.615605 10836 raft_consensus.cc:1270] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:50.617444 11148 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:50.633868 10911 raft_consensus.cc:1270] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:56:50.636637 11130 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:56:50.638679 10911 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:56:50.643612 11130 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250113 08:56:50.896445 10933 debug-util.cc:398] Leaking SignalData structure 0x7b0800131060 after lost signal to thread 10705
W20250113 08:56:50.897178 10933 debug-util.cc:398] Leaking SignalData structure 0x7b0800131740 after lost signal to thread 10768
W20250113 08:56:50.897958 10933 debug-util.cc:398] Leaking SignalData structure 0x7b08001e27a0 after lost signal to thread 10861
W20250113 08:56:50.898635 10933 debug-util.cc:398] Leaking SignalData structure 0x7b08000423e0 after lost signal to thread 10936
W20250113 08:56:50.899307 10933 debug-util.cc:398] Leaking SignalData structure 0x7b08000400a0 after lost signal to thread 11010
W20250113 08:56:50.899587 10933 debug-util.cc:398] Leaking SignalData structure 0x7b0800041a80 after lost signal to thread 11131
W20250113 08:56:50.899767 10933 debug-util.cc:398] Leaking SignalData structure 0x7b0800041760 after lost signal to thread 11133
I20250113 08:56:51.105516 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:51.113989 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:51.114542 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:52.136947 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:52.154428 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:52.155134 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:53.156138 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:53.160800 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:53.161216 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:54.162202 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:54.167085 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:54.167507 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:55.168185 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:55.172752 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:55.173138 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:56.173887 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:56.178467 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:56.178864 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:57.179715 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:57.184410 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:57.184787 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:58.185580 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:58.190186 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:58.190554 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:56:59.191310 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:56:59.195917 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:56:59.196290 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:00.220963 10702 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20250113 08:57:00.225132 10833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
mode: GRACEFUL
new_leader_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:34712
I20250113 08:57:00.225907 10833 raft_consensus.cc:604] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Received request to transfer leadership to TS 91fbac8f141842c6803e02488ae2e454
I20250113 08:57:00.230293 10833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
mode: GRACEFUL
new_leader_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:34712
I20250113 08:57:00.230831 10833 raft_consensus.cc:604] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Received request to transfer leadership to TS 91fbac8f141842c6803e02488ae2e454
I20250113 08:57:00.235147 10986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
mode: GRACEFUL
new_leader_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:35196
I20250113 08:57:00.235745 10986 raft_consensus.cc:604] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 LEADER]: Received request to transfer leadership to TS 91fbac8f141842c6803e02488ae2e454
I20250113 08:57:00.240139 10833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
mode: GRACEFUL
new_leader_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:34712
I20250113 08:57:00.240731 10833 raft_consensus.cc:604] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Received request to transfer leadership to TS 91fbac8f141842c6803e02488ae2e454
I20250113 08:57:00.247238 10986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
mode: GRACEFUL
new_leader_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:35196
I20250113 08:57:00.247658 10986 raft_consensus.cc:604] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 LEADER]: Received request to transfer leadership to TS 91fbac8f141842c6803e02488ae2e454
I20250113 08:57:00.513386 11135 raft_consensus.cc:988] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: : Instructing follower 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.513906 11164 raft_consensus.cc:1076] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 LEADER]: Signalling peer 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.515348 10911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:50818
I20250113 08:57:00.515779 10911 raft_consensus.cc:491] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:00.516119 10911 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:00.520725 10911 raft_consensus.cc:513] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.522763 10911 leader_election.cc:290] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 4 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:57:00.523785 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:00.523785 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:57:00.524590 10986 raft_consensus.cc:3049] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:00.524644 10833 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:00.524885 10986 raft_consensus.cc:738] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:57:00.525614 10986 consensus_queue.cc:260] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.526556 10986 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:00.529910 10833 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 4.
I20250113 08:57:00.530859 10986 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 4.
I20250113 08:57:00.531144 10875 leader_election.cc:304] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:57:00.532040 11112 raft_consensus.cc:2798] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:00.532744 11112 raft_consensus.cc:695] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:00.533495 11112 consensus_queue.cc:237] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.541303 10734 catalog_manager.cc:5526] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 3 to 4, leader changed from 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131) to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 4 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:00.577632 11162 raft_consensus.cc:988] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b: : Instructing follower 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.578130 11171 raft_consensus.cc:1076] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Signalling peer 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.579456 10911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:50812
I20250113 08:57:00.579864 10911 raft_consensus.cc:491] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:00.580112 10911 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.584051 10911 raft_consensus.cc:513] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.585620 10911 leader_election.cc:290] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:57:00.586318 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:57:00.586894 10833 raft_consensus.cc:3049] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:00.586881 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:00.587352 10833 raft_consensus.cc:738] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:57:00.587524 10986 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.588117 10833 consensus_queue.cc:260] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.589531 10833 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.593982 10986 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.595170 10876 leader_election.cc:304] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:57:00.596031 11112 raft_consensus.cc:2798] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:00.596061 10833 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.596493 11112 raft_consensus.cc:695] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:00.597389 11112 consensus_queue.cc:237] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.604053 10734 catalog_manager.cc:5526] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 2 to 3, leader changed from 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129) to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 3 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:00.612270 11135 raft_consensus.cc:988] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64: : Instructing follower 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.612668 11164 raft_consensus.cc:1076] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 LEADER]: Signalling peer 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.613940 10911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:50818
I20250113 08:57:00.614356 10911 raft_consensus.cc:491] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:00.614624 10911 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.618877 10911 raft_consensus.cc:513] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.620308 10911 leader_election.cc:290] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:57:00.621215 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:57:00.621337 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:00.621784 10986 raft_consensus.cc:3049] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:00.621748 10833 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.622161 10986 raft_consensus.cc:738] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:57:00.622712 10986 consensus_queue.cc:260] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.623544 10986 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.626673 10833 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.627606 10986 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.627604 10875 leader_election.cc:304] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:57:00.628480 11112 raft_consensus.cc:2798] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:00.628922 11112 raft_consensus.cc:695] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:00.629592 11112 consensus_queue.cc:237] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.631980 11162 raft_consensus.cc:988] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b: : Instructing follower 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.632469 11171 raft_consensus.cc:1076] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Signalling peer 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.633922 10911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:50812
I20250113 08:57:00.634387 10911 raft_consensus.cc:491] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:00.634661 10911 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.637063 10734 catalog_manager.cc:5526] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 2 to 3, leader changed from 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131) to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 3 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:00.639394 10911 raft_consensus.cc:513] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:57:00.641280 10911 leader_election.cc:290] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Requested vote from peers 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571), 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529)
I20250113 08:57:00.642025 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:00.642288 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c6dad88d334aeebecc08f18f4ec785" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:57:00.642583 10986 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.642861 10833 raft_consensus.cc:3049] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:00.643167 10833 raft_consensus.cc:738] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:57:00.643982 10833 consensus_queue.cc:260] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:57:00.645270 10833 raft_consensus.cc:3054] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.648146 11162 raft_consensus.cc:988] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b: : Instructing follower 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.648628 11171 raft_consensus.cc:1076] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Signalling peer 91fbac8f141842c6803e02488ae2e454 to start an election
I20250113 08:57:00.648599 10986 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.649823 10876 leader_election.cc:304] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:57:00.650053 10911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
from {username='slave'} at 127.0.0.1:50812
I20250113 08:57:00.650521 10911 raft_consensus.cc:491] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:00.650640 11112 raft_consensus.cc:2798] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:00.650879 10911 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.651265 10833 raft_consensus.cc:2463] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.651070 11112 raft_consensus.cc:695] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:00.652349 11112 consensus_queue.cc:237] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } }
I20250113 08:57:00.656981 10911 raft_consensus.cc:513] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.658468 10911 leader_election.cc:290] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:57:00.659572 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:57:00.659641 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "91fbac8f141842c6803e02488ae2e454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:00.660087 10833 raft_consensus.cc:3049] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:00.660210 10986 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.660463 10833 raft_consensus.cc:738] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:57:00.660017 10734 catalog_manager.cc:5526] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 2 to 3, leader changed from 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129) to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 3 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:00.661185 10833 consensus_queue.cc:260] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.662213 10833 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:00.665027 10986 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.666247 10876 leader_election.cc:304] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6996c7a04c4c3bb45b5defaa8cab64, 91fbac8f141842c6803e02488ae2e454; no voters:
I20250113 08:57:00.667156 11112 raft_consensus.cc:2798] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:00.667591 11112 raft_consensus.cc:695] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Becoming Leader. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:00.668540 10833 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91fbac8f141842c6803e02488ae2e454 in term 3.
I20250113 08:57:00.668416 11112 consensus_queue.cc:237] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:00.674845 10734 catalog_manager.cc:5526] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 reported cstate change: term changed from 2 to 3, leader changed from 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129) to 91fbac8f141842c6803e02488ae2e454 (127.10.115.130). New cstate: current_term: 3 leader_uuid: "91fbac8f141842c6803e02488ae2e454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:00.988453 10833 raft_consensus.cc:1270] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:00.989691 11165 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:00.999759 10986 raft_consensus.cc:1270] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:01.001783 11165 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:57:01.014200 10983 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:01.015524 11179 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:57:01.024641 10833 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:01.026028 11179 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:57:01.074815 10983 raft_consensus.cc:1270] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:01.075979 11179 consensus_queue.cc:1035] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:01.085502 10833 raft_consensus.cc:1270] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:01.086871 11179 consensus_queue.cc:1035] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:01.107792 10983 raft_consensus.cc:1270] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:01.109129 11179 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:01.117007 10833 raft_consensus.cc:1270] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:01.118423 11112 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:01.136747 10833 raft_consensus.cc:1270] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:01.137861 11112 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:01.145311 10986 raft_consensus.cc:1270] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Refusing update from remote peer 91fbac8f141842c6803e02488ae2e454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:01.147217 11112 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:03.281816 10702 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20250113 08:57:03.282598 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:03.289510 10911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
mode: GRACEFUL
new_leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:50820
I20250113 08:57:03.290359 10911 raft_consensus.cc:604] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Received request to transfer leadership to TS 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:57:03.292385 10911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
mode: GRACEFUL
new_leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:50820
I20250113 08:57:03.292790 10911 raft_consensus.cc:604] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 LEADER]: Received request to transfer leadership to TS 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:57:03.294550 10911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
mode: GRACEFUL
new_leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:50820
I20250113 08:57:03.294926 10911 raft_consensus.cc:604] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Received request to transfer leadership to TS 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:57:03.296716 10911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
mode: GRACEFUL
new_leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:50820
I20250113 08:57:03.297137 10911 raft_consensus.cc:604] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 LEADER]: Received request to transfer leadership to TS 2e6996c7a04c4c3bb45b5defaa8cab64
I20250113 08:57:03.298975 10911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929"
dest_uuid: "91fbac8f141842c6803e02488ae2e454"
mode: GRACEFUL
new_leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:50820
I20250113 08:57:03.299350 10911 raft_consensus.cc:604] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Received request to transfer leadership to TS 02844150e71e4ea7b8d3c5069b179e6b
I20250113 08:57:03.300127 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250113 08:57:03.300529 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:03.417934 11179 raft_consensus.cc:988] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454: : Instructing follower 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:57:03.418453 11191 raft_consensus.cc:1076] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Signalling peer 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:57:03.419926 10835 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:34704
I20250113 08:57:03.420397 10835 raft_consensus.cc:491] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:03.420858 10835 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.426579 10835 raft_consensus.cc:513] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.428746 10835 leader_election.cc:290] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 4 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:57:03.429997 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:03.430539 10986 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.430709 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4bc430a35487c8c06b493400b352c" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:57:03.431152 10911 raft_consensus.cc:3049] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:03.431356 10911 raft_consensus.cc:738] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:03.431927 10911 consensus_queue.cc:260] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.432862 10911 raft_consensus.cc:3054] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.435222 10986 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 4.
I20250113 08:57:03.436407 10801 leader_election.cc:304] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:57:03.437747 10911 raft_consensus.cc:2463] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 4.
I20250113 08:57:03.438024 11193 raft_consensus.cc:2798] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:03.439612 11193 raft_consensus.cc:695] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 4 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:57:03.441021 11193 consensus_queue.cc:237] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.447466 10731 catalog_manager.cc:5526] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 3 to 4, leader changed from 91fbac8f141842c6803e02488ae2e454 (127.10.115.130) to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 4 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:03.543354 11191 raft_consensus.cc:988] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454: : Instructing follower 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:57:03.543810 11179 raft_consensus.cc:1076] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 LEADER]: Signalling peer 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:57:03.545079 10986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:35192
I20250113 08:57:03.545519 10986 raft_consensus.cc:491] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:03.545773 10986 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:03.549968 10986 raft_consensus.cc:513] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.551548 10986 leader_election.cc:290] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 3 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:57:03.552491 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:57:03.552395 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27d8218f7154005b6e5829d2567db6b" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:57:03.553014 10833 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:03.553089 10911 raft_consensus.cc:3049] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:03.553432 10911 raft_consensus.cc:738] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:03.554056 10911 consensus_queue.cc:260] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.555294 10911 raft_consensus.cc:3054] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:03.559176 10833 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 3.
I20250113 08:57:03.559962 10911 raft_consensus.cc:2463] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 3.
I20250113 08:57:03.560158 10949 leader_election.cc:304] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:57:03.560945 11192 raft_consensus.cc:2798] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:03.561967 11192 raft_consensus.cc:695] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:57:03.562774 11192 consensus_queue.cc:237] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.568928 10731 catalog_manager.cc:5526] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 2 to 3, leader changed from 91fbac8f141842c6803e02488ae2e454 (127.10.115.130) to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 3 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:03.624801 11191 raft_consensus.cc:988] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: : Instructing follower 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:57:03.625170 11179 raft_consensus.cc:1076] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Signalling peer 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:57:03.626578 10833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:34704
I20250113 08:57:03.626971 10833 raft_consensus.cc:491] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:03.627219 10833 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.631232 10833 raft_consensus.cc:513] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.632838 10833 leader_election.cc:290] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 4 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:57:03.633564 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:57:03.634111 10911 raft_consensus.cc:3049] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:03.634232 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0382b50b8cbb4b26ad00df2a811426d6" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:03.634413 10911 raft_consensus.cc:738] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:03.634816 10986 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.635233 10911 consensus_queue.cc:260] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.636374 10911 raft_consensus.cc:3054] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.640865 10986 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 4.
I20250113 08:57:03.641839 10801 leader_election.cc:304] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:57:03.642035 10911 raft_consensus.cc:2463] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 4.
I20250113 08:57:03.642566 11193 raft_consensus.cc:2798] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:03.642997 11193 raft_consensus.cc:695] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:57:03.643679 11193 consensus_queue.cc:237] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.650244 10731 catalog_manager.cc:5526] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 3 to 4, leader changed from 91fbac8f141842c6803e02488ae2e454 (127.10.115.130) to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 4 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:03.681188 11191 raft_consensus.cc:988] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454: : Instructing follower 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:57:03.682092 11179 raft_consensus.cc:1076] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 LEADER]: Signalling peer 2e6996c7a04c4c3bb45b5defaa8cab64 to start an election
I20250113 08:57:03.683636 10983 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1edf48023a9b41159bb90511619ef496"
dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
from {username='slave'} at 127.0.0.1:35192
I20250113 08:57:03.684095 10983 raft_consensus.cc:491] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:03.684376 10983 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:03.689116 10983 raft_consensus.cc:513] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.690640 10983 leader_election.cc:290] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 5 election: Requested vote from peers 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529), 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067)
I20250113 08:57:03.691557 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:57:03.691809 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf48023a9b41159bb90511619ef496" candidate_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
I20250113 08:57:03.692174 10911 raft_consensus.cc:3049] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:57:03.692464 10911 raft_consensus.cc:738] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:03.692344 10833 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:03.693158 10911 consensus_queue.cc:260] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.694324 10911 raft_consensus.cc:3054] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:03.698496 10833 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 5.
I20250113 08:57:03.699013 10911 raft_consensus.cc:2463] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6996c7a04c4c3bb45b5defaa8cab64 in term 5.
I20250113 08:57:03.699406 10949 leader_election.cc:304] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:57:03.700172 11192 raft_consensus.cc:2798] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:57:03.700531 11192 raft_consensus.cc:695] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 5 LEADER]: Becoming Leader. State: Replica: 2e6996c7a04c4c3bb45b5defaa8cab64, State: Running, Role: LEADER
I20250113 08:57:03.701316 11192 consensus_queue.cc:237] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.707619 10731 catalog_manager.cc:5526] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 reported cstate change: term changed from 4 to 5, leader changed from 91fbac8f141842c6803e02488ae2e454 (127.10.115.130) to 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131). New cstate: current_term: 5 leader_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:03.897485 11203 raft_consensus.cc:988] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454: : Instructing follower 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:57:03.897913 11179 raft_consensus.cc:1076] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Signalling peer 02844150e71e4ea7b8d3c5069b179e6b to start an election
I20250113 08:57:03.899191 10833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929"
dest_uuid: "02844150e71e4ea7b8d3c5069b179e6b"
from {username='slave'} at 127.0.0.1:34704
I20250113 08:57:03.899608 10833 raft_consensus.cc:491] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:03.899855 10833 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.903927 10833 raft_consensus.cc:513] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.905301 10833 leader_election.cc:290] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 4 election: Requested vote from peers 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067), 2e6996c7a04c4c3bb45b5defaa8cab64 (127.10.115.131:41571)
I20250113 08:57:03.906272 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64"
I20250113 08:57:03.906735 10983 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.906756 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbed4a53ed94dd4a54927a541a11929" candidate_uuid: "02844150e71e4ea7b8d3c5069b179e6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "91fbac8f141842c6803e02488ae2e454"
I20250113 08:57:03.907286 10911 raft_consensus.cc:3049] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:03.907503 10911 raft_consensus.cc:738] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 91fbac8f141842c6803e02488ae2e454, State: Running, Role: LEADER
I20250113 08:57:03.908067 10911 consensus_queue.cc:260] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.908942 10911 raft_consensus.cc:3054] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:03.912184 10983 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 4.
I20250113 08:57:03.913077 10801 leader_election.cc:304] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02844150e71e4ea7b8d3c5069b179e6b, 2e6996c7a04c4c3bb45b5defaa8cab64; no voters:
I20250113 08:57:03.913848 11193 raft_consensus.cc:2798] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:03.914139 10911 raft_consensus.cc:2463] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02844150e71e4ea7b8d3c5069b179e6b in term 4.
I20250113 08:57:03.914278 11193 raft_consensus.cc:695] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 LEADER]: Becoming Leader. State: Replica: 02844150e71e4ea7b8d3c5069b179e6b, State: Running, Role: LEADER
I20250113 08:57:03.915254 11193 consensus_queue.cc:237] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } }
I20250113 08:57:03.919176 10983 raft_consensus.cc:1270] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:03.920373 11193 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:03.922307 10731 catalog_manager.cc:5526] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b reported cstate change: term changed from 3 to 4, leader changed from 91fbac8f141842c6803e02488ae2e454 (127.10.115.130) to 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129). New cstate: current_term: 4 leader_uuid: "02844150e71e4ea7b8d3c5069b179e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:03.934154 10911 raft_consensus.cc:1270] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:03.936349 11193 consensus_queue.cc:1035] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:03.991629 10833 raft_consensus.cc:1270] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:03.992794 11192 consensus_queue.cc:1035] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:04.000624 10911 raft_consensus.cc:1270] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:04.002406 11200 consensus_queue.cc:1035] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:57:04.032938 10911 raft_consensus.cc:1270] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:04.034096 11209 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:04.042797 10983 raft_consensus.cc:1270] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:04.044121 11209 consensus_queue.cc:1035] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:04.108191 10833 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 5 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250113 08:57:04.109395 11200 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02844150e71e4ea7b8d3c5069b179e6b" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:57:04.117609 10911 raft_consensus.cc:1270] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 5 FOLLOWER]: Refusing update from remote peer 2e6996c7a04c4c3bb45b5defaa8cab64: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250113 08:57:04.119215 11192 consensus_queue.cc:1035] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:57:04.301369 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:04.306000 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:04.306365 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:04.348560 10983 raft_consensus.cc:1270] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:04.349598 11209 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6996c7a04c4c3bb45b5defaa8cab64" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:04.357158 10911 raft_consensus.cc:1270] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Refusing update from remote peer 02844150e71e4ea7b8d3c5069b179e6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:04.358566 11209 consensus_queue.cc:1035] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fbac8f141842c6803e02488ae2e454" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:05.307184 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:05.312074 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:05.312479 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:06.313186 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:06.318011 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:06.318413 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:07.319144 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:07.323935 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:07.324332 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:08.325186 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:08.330386 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:08.330890 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:09.331665 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:09.336302 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:09.336681 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:10.337494 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:10.342186 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:10.342638 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:11.343336 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:11.348488 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:11.348914 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:12.353206 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:12.362669 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:12.363246 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:13.388366 10702 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20250113 08:57:13.389024 10702 tablet_server.cc:178] TabletServer@127.10.115.129:0 shutting down...
I20250113 08:57:13.420140 10702 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 08:57:13.420840 10702 tablet_replica.cc:331] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.421834 10702 raft_consensus.cc:2238] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.423051 10702 raft_consensus.cc:2267] T dbbed4a53ed94dd4a54927a541a11929 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.425843 10702 tablet_replica.cc:331] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.426309 10702 raft_consensus.cc:2238] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.426707 10702 raft_consensus.cc:2267] T d27d8218f7154005b6e5829d2567db6b P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.428215 10702 tablet_replica.cc:331] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.428679 10702 raft_consensus.cc:2238] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 4 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.429522 10702 raft_consensus.cc:2267] T 8ca4bc430a35487c8c06b493400b352c P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.431085 10702 tablet_replica.cc:331] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.431511 10702 raft_consensus.cc:2238] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.432292 10702 raft_consensus.cc:2267] T 0382b50b8cbb4b26ad00df2a811426d6 P 02844150e71e4ea7b8d3c5069b179e6b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.434279 10702 tablet_replica.cc:331] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.434733 10702 raft_consensus.cc:2238] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 5 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.435128 10702 raft_consensus.cc:2267] T 1edf48023a9b41159bb90511619ef496 P 02844150e71e4ea7b8d3c5069b179e6b [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.436609 10702 tablet_replica.cc:331] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.437067 10702 raft_consensus.cc:2238] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.437475 10702 raft_consensus.cc:2267] T b8e5506e1cd340f0a7e323883c6e2e61 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.439041 10702 tablet_replica.cc:331] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.439513 10702 raft_consensus.cc:2238] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.439903 10702 raft_consensus.cc:2267] T b4c6dad88d334aeebecc08f18f4ec785 P 02844150e71e4ea7b8d3c5069b179e6b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.441596 10702 tablet_replica.cc:331] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.442031 10702 raft_consensus.cc:2238] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.442435 10702 raft_consensus.cc:2267] T 53b97192388a41b6881eef9331803c23 P 02844150e71e4ea7b8d3c5069b179e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.444001 10702 tablet_replica.cc:331] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b: stopping tablet replica
I20250113 08:57:13.444412 10702 raft_consensus.cc:2238] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.444784 10702 raft_consensus.cc:2267] T 000d4553694048038746f484781f7e33 P 02844150e71e4ea7b8d3c5069b179e6b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.525951 10702 tablet_server.cc:195] TabletServer@127.10.115.129:0 shutdown complete.
I20250113 08:57:13.544600 10702 tablet_server.cc:178] TabletServer@127.10.115.130:0 shutting down...
W20250113 08:57:13.556929 10875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.115.129:37529: connect: Connection refused (error 111)
W20250113 08:57:13.561344 10875 consensus_peers.cc:487] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 -> Peer 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529): Couldn't send request to peer 02844150e71e4ea7b8d3c5069b179e6b. Status: Network error: Client connection negotiation failed: client connection to 127.10.115.129:37529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:13.575274 10702 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 08:57:13.575994 10702 tablet_replica.cc:331] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.576635 10702 raft_consensus.cc:2238] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.577090 10702 raft_consensus.cc:2267] T 000d4553694048038746f484781f7e33 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.578886 10702 tablet_replica.cc:331] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.579391 10702 raft_consensus.cc:2238] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.579769 10702 raft_consensus.cc:2267] T dbbed4a53ed94dd4a54927a541a11929 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.581326 10702 tablet_replica.cc:331] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.581789 10702 raft_consensus.cc:2238] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.582190 10702 raft_consensus.cc:2267] T d27d8218f7154005b6e5829d2567db6b P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.583760 10702 tablet_replica.cc:331] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.584233 10702 raft_consensus.cc:2238] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.584612 10702 raft_consensus.cc:2267] T 0382b50b8cbb4b26ad00df2a811426d6 P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.586155 10702 tablet_replica.cc:331] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.586589 10702 raft_consensus.cc:2238] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.586956 10702 raft_consensus.cc:2267] T 1edf48023a9b41159bb90511619ef496 P 91fbac8f141842c6803e02488ae2e454 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.588536 10702 tablet_replica.cc:331] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.589035 10702 raft_consensus.cc:2238] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 1 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.589789 10702 raft_consensus.cc:2267] T 53b97192388a41b6881eef9331803c23 P 91fbac8f141842c6803e02488ae2e454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.591452 10702 tablet_replica.cc:331] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.591914 10702 raft_consensus.cc:2238] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 2 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.592656 10702 raft_consensus.cc:2267] T b8e5506e1cd340f0a7e323883c6e2e61 P 91fbac8f141842c6803e02488ae2e454 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.594672 10702 tablet_replica.cc:331] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.595264 10702 raft_consensus.cc:2238] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 3 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.596182 10702 raft_consensus.cc:2267] T b4c6dad88d334aeebecc08f18f4ec785 P 91fbac8f141842c6803e02488ae2e454 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.597980 10702 tablet_replica.cc:331] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454: stopping tablet replica
I20250113 08:57:13.598577 10702 raft_consensus.cc:2238] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.599115 10702 raft_consensus.cc:2267] T 8ca4bc430a35487c8c06b493400b352c P 91fbac8f141842c6803e02488ae2e454 [term 4 FOLLOWER]: Raft consensus is shut down!
W20250113 08:57:13.627637 10949 consensus_peers.cc:487] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 -> Peer 02844150e71e4ea7b8d3c5069b179e6b (127.10.115.129:37529): Couldn't send request to peer 02844150e71e4ea7b8d3c5069b179e6b. Status: Network error: Client connection negotiation failed: client connection to 127.10.115.129:37529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:13.645133 10702 tablet_server.cc:195] TabletServer@127.10.115.130:0 shutdown complete.
I20250113 08:57:13.662858 10702 tablet_server.cc:178] TabletServer@127.10.115.131:0 shutting down...
W20250113 08:57:13.679876 10950 consensus_peers.cc:487] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 -> Peer 91fbac8f141842c6803e02488ae2e454 (127.10.115.130:43067): Couldn't send request to peer 91fbac8f141842c6803e02488ae2e454. Status: Network error: Client connection negotiation failed: client connection to 127.10.115.130:43067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:13.693504 10702 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 08:57:13.694200 10702 tablet_replica.cc:331] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.694821 10702 raft_consensus.cc:2238] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.695340 10702 raft_consensus.cc:2267] T 8ca4bc430a35487c8c06b493400b352c P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.697062 10702 tablet_replica.cc:331] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.697567 10702 raft_consensus.cc:2238] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.698278 10702 raft_consensus.cc:2267] T d27d8218f7154005b6e5829d2567db6b P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.700210 10702 tablet_replica.cc:331] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.700677 10702 raft_consensus.cc:2238] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.701038 10702 raft_consensus.cc:2267] T 53b97192388a41b6881eef9331803c23 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.702653 10702 tablet_replica.cc:331] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.703119 10702 raft_consensus.cc:2238] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 5 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.703893 10702 raft_consensus.cc:2267] T 1edf48023a9b41159bb90511619ef496 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.705571 10702 tablet_replica.cc:331] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.706028 10702 raft_consensus.cc:2238] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.706422 10702 raft_consensus.cc:2267] T 0382b50b8cbb4b26ad00df2a811426d6 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.708076 10702 tablet_replica.cc:331] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.708519 10702 raft_consensus.cc:2238] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.708942 10702 raft_consensus.cc:2267] T b8e5506e1cd340f0a7e323883c6e2e61 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.710487 10702 tablet_replica.cc:331] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.710919 10702 raft_consensus.cc:2238] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.711326 10702 raft_consensus.cc:2267] T b4c6dad88d334aeebecc08f18f4ec785 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.712749 10702 tablet_replica.cc:331] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.713198 10702 raft_consensus.cc:2238] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:13.713657 10702 raft_consensus.cc:2267] T dbbed4a53ed94dd4a54927a541a11929 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.715091 10702 tablet_replica.cc:331] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64: stopping tablet replica
I20250113 08:57:13.715480 10702 raft_consensus.cc:2238] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.716212 10702 raft_consensus.cc:2267] T 000d4553694048038746f484781f7e33 P 2e6996c7a04c4c3bb45b5defaa8cab64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.742573 10702 tablet_server.cc:195] TabletServer@127.10.115.131:0 shutdown complete.
I20250113 08:57:13.762460 10702 master.cc:537] Master@127.10.115.190:46767 shutting down...
I20250113 08:57:13.885517 10702 raft_consensus.cc:2238] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 1 LEADER]: Raft consensus shutting down.
I20250113 08:57:13.886160 10702 raft_consensus.cc:2267] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:13.886484 10702 tablet_replica.cc:331] T 00000000000000000000000000000000 P de95065bb0b64f679d49107727bb2808: stopping tablet replica
I20250113 08:57:13.906541 10702 master.cc:559] Master@127.10.115.190:46767 shutdown complete.
02844150e71e4ea7b8d3c5069b179e6b 4
2e6996c7a04c4c3bb45b5defaa8cab64 1
91fbac8f141842c6803e02488ae2e454 4
02844150e71e4ea7b8d3c5069b179e6b 3
2e6996c7a04c4c3bb45b5defaa8cab64 3
91fbac8f141842c6803e02488ae2e454 3
02844150e71e4ea7b8d3c5069b179e6b 0
2e6996c7a04c4c3bb45b5defaa8cab64 1
91fbac8f141842c6803e02488ae2e454 8
02844150e71e4ea7b8d3c5069b179e6b 3
2e6996c7a04c4c3bb45b5defaa8cab64 3
91fbac8f141842c6803e02488ae2e454 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31847 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250113 08:57:13.946002 10702 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.115.190:41507
I20250113 08:57:13.947083 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:13.953599 11232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:13.954699 11233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:13.955714 11235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:13.956547 10702 server_base.cc:1034] running on GCE node
I20250113 08:57:13.957438 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:13.957657 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:13.957769 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758633957759 us; error 0 us; skew 500 ppm
I20250113 08:57:13.958220 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:13.960575 10702 webserver.cc:458] Webserver started at http://127.10.115.190:36695/ using document root <none> and password file <none>
I20250113 08:57:13.961058 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:13.961232 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:13.961567 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:13.962704 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/master-0-root/instance:
uuid: "daca8b8de715409bb686ad7e2f5c98ae"
format_stamp: "Formatted at 2025-01-13 08:57:13 on dist-test-slave-lrhs"
I20250113 08:57:13.967135 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250113 08:57:13.970520 11240 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:13.971307 10702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250113 08:57:13.971558 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/master-0-root
uuid: "daca8b8de715409bb686ad7e2f5c98ae"
format_stamp: "Formatted at 2025-01-13 08:57:13 on dist-test-slave-lrhs"
I20250113 08:57:13.971822 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:13.990712 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:13.991966 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:14.029011 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.190:41507
I20250113 08:57:14.029099 11291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.190:41507 every 8 connection(s)
I20250113 08:57:14.032946 11292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.042517 11292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae: Bootstrap starting.
I20250113 08:57:14.047063 11292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.051132 11292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae: No bootstrap required, opened a new log
I20250113 08:57:14.053148 11292 raft_consensus.cc:357] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daca8b8de715409bb686ad7e2f5c98ae" member_type: VOTER }
I20250113 08:57:14.053615 11292 raft_consensus.cc:383] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.053843 11292 raft_consensus.cc:738] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daca8b8de715409bb686ad7e2f5c98ae, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.054416 11292 consensus_queue.cc:260] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daca8b8de715409bb686ad7e2f5c98ae" member_type: VOTER }
I20250113 08:57:14.054843 11292 raft_consensus.cc:397] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 08:57:14.055070 11292 raft_consensus.cc:491] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 08:57:14.055327 11292 raft_consensus.cc:3054] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:14.059130 11292 raft_consensus.cc:513] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daca8b8de715409bb686ad7e2f5c98ae" member_type: VOTER }
I20250113 08:57:14.059602 11292 leader_election.cc:304] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daca8b8de715409bb686ad7e2f5c98ae; no voters:
I20250113 08:57:14.060678 11292 leader_election.cc:290] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 08:57:14.061038 11295 raft_consensus.cc:2798] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:14.062311 11295 raft_consensus.cc:695] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 1 LEADER]: Becoming Leader. State: Replica: daca8b8de715409bb686ad7e2f5c98ae, State: Running, Role: LEADER
I20250113 08:57:14.062940 11295 consensus_queue.cc:237] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daca8b8de715409bb686ad7e2f5c98ae" member_type: VOTER }
I20250113 08:57:14.063575 11292 sys_catalog.cc:564] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [sys.catalog]: configured and running, proceeding with master startup.
I20250113 08:57:14.068310 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader daca8b8de715409bb686ad7e2f5c98ae. Latest consensus state: current_term: 1 leader_uuid: "daca8b8de715409bb686ad7e2f5c98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daca8b8de715409bb686ad7e2f5c98ae" member_type: VOTER } }
I20250113 08:57:14.068351 11296 sys_catalog.cc:455] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daca8b8de715409bb686ad7e2f5c98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daca8b8de715409bb686ad7e2f5c98ae" member_type: VOTER } }
I20250113 08:57:14.069281 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [sys.catalog]: This master's current role is: LEADER
I20250113 08:57:14.069439 11296 sys_catalog.cc:458] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [sys.catalog]: This master's current role is: LEADER
I20250113 08:57:14.075033 11302 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 08:57:14.083083 11302 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 08:57:14.084069 10702 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 08:57:14.092144 11302 catalog_manager.cc:1348] Generated new cluster ID: e2d944c9fad54e8d82db27c92347a6e8
I20250113 08:57:14.092463 11302 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 08:57:14.106607 11302 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 08:57:14.107836 11302 catalog_manager.cc:1505] Loading token signing keys...
I20250113 08:57:14.123566 11302 catalog_manager.cc:5899] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae: Generated new TSK 0
I20250113 08:57:14.124236 11302 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 08:57:14.150691 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:14.157343 11313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:14.158221 11314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:14.159205 11316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:14.160192 10702 server_base.cc:1034] running on GCE node
I20250113 08:57:14.161005 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:14.161182 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:14.161348 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758634161328 us; error 0 us; skew 500 ppm
I20250113 08:57:14.161799 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:14.164014 10702 webserver.cc:458] Webserver started at http://127.10.115.129:36343/ using document root <none> and password file <none>
I20250113 08:57:14.164460 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:14.164611 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:14.164860 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:14.165946 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-0-root/instance:
uuid: "b793334f8c5a4a6aa947ec73fe474b71"
format_stamp: "Formatted at 2025-01-13 08:57:14 on dist-test-slave-lrhs"
I20250113 08:57:14.170260 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20250113 08:57:14.173223 11321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.174104 10702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250113 08:57:14.174392 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-0-root
uuid: "b793334f8c5a4a6aa947ec73fe474b71"
format_stamp: "Formatted at 2025-01-13 08:57:14 on dist-test-slave-lrhs"
I20250113 08:57:14.174695 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:14.192174 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:14.193378 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:14.194711 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:57:14.196926 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:57:14.197103 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.197373 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:57:14.197525 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.236792 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.129:41471
I20250113 08:57:14.236884 11383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.129:41471 every 8 connection(s)
I20250113 08:57:14.241716 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:14.250432 11388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:14.251529 11389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:14.253866 11384 heartbeater.cc:346] Connected to a master server at 127.10.115.190:41507
I20250113 08:57:14.254222 11384 heartbeater.cc:463] Registering TS with master...
I20250113 08:57:14.254930 11384 heartbeater.cc:510] Master 127.10.115.190:41507 requested a full tablet report, sending...
I20250113 08:57:14.255277 10702 server_base.cc:1034] running on GCE node
W20250113 08:57:14.256412 11391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:14.257229 11257 ts_manager.cc:194] Registered new tserver with Master: b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471)
I20250113 08:57:14.257409 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:14.257673 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:14.257870 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758634257853 us; error 0 us; skew 500 ppm
I20250113 08:57:14.258615 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:14.259603 11257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54020
I20250113 08:57:14.261512 10702 webserver.cc:458] Webserver started at http://127.10.115.130:43561/ using document root <none> and password file <none>
I20250113 08:57:14.262179 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:14.262419 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:14.262645 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:14.263728 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-1-root/instance:
uuid: "ea0cbc9eb22d4063b00e12349d64f478"
format_stamp: "Formatted at 2025-01-13 08:57:14 on dist-test-slave-lrhs"
I20250113 08:57:14.268189 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250113 08:57:14.271494 11396 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.272269 10702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250113 08:57:14.272547 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-1-root
uuid: "ea0cbc9eb22d4063b00e12349d64f478"
format_stamp: "Formatted at 2025-01-13 08:57:14 on dist-test-slave-lrhs"
I20250113 08:57:14.272801 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:14.289239 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:14.290524 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:14.291980 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:57:14.294173 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:57:14.294365 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.294586 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:57:14.294734 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.334739 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.130:42101
I20250113 08:57:14.334846 11458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.130:42101 every 8 connection(s)
I20250113 08:57:14.339501 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:14.347246 11462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:14.348754 11463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:14.350253 10702 server_base.cc:1034] running on GCE node
I20250113 08:57:14.351235 11459 heartbeater.cc:346] Connected to a master server at 127.10.115.190:41507
I20250113 08:57:14.351610 11459 heartbeater.cc:463] Registering TS with master...
W20250113 08:57:14.351584 11465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:14.352475 11459 heartbeater.cc:510] Master 127.10.115.190:41507 requested a full tablet report, sending...
I20250113 08:57:14.352602 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:14.352874 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:14.353021 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758634353008 us; error 0 us; skew 500 ppm
I20250113 08:57:14.353614 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:14.354650 11257 ts_manager.cc:194] Registered new tserver with Master: ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:14.355968 11257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54034
I20250113 08:57:14.356547 10702 webserver.cc:458] Webserver started at http://127.10.115.131:42715/ using document root <none> and password file <none>
I20250113 08:57:14.357129 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:14.357363 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:14.357617 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:14.358755 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-2-root/instance:
uuid: "10711487b8dc43f181e340bd29bc2327"
format_stamp: "Formatted at 2025-01-13 08:57:14 on dist-test-slave-lrhs"
I20250113 08:57:14.363170 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250113 08:57:14.366516 11470 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.367254 10702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250113 08:57:14.367520 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-2-root
uuid: "10711487b8dc43f181e340bd29bc2327"
format_stamp: "Formatted at 2025-01-13 08:57:14 on dist-test-slave-lrhs"
I20250113 08:57:14.367774 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736758601939195-10702-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:14.379624 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:14.380807 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:14.382333 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:57:14.384567 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:57:14.384763 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.385004 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:57:14.385166 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:14.424382 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.131:41233
I20250113 08:57:14.424474 11532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.131:41233 every 8 connection(s)
I20250113 08:57:14.437606 11533 heartbeater.cc:346] Connected to a master server at 127.10.115.190:41507
I20250113 08:57:14.437994 11533 heartbeater.cc:463] Registering TS with master...
I20250113 08:57:14.438690 11533 heartbeater.cc:510] Master 127.10.115.190:41507 requested a full tablet report, sending...
I20250113 08:57:14.440603 11257 ts_manager.cc:194] Registered new tserver with Master: 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:14.440760 10702 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01343327s
I20250113 08:57:14.442425 11257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54040
I20250113 08:57:14.443872 10702 test_util.cc:274] Using random seed: -1586006973
I20250113 08:57:14.465691 11257 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54056:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 08:57:14.468570 11257 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 08:57:14.535683 11497 tablet_service.cc:1467] Processing CreateTablet for tablet 394f7fa139c04737ae3f52f53323923f (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250113 08:57:14.537215 11497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394f7fa139c04737ae3f52f53323923f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.535688 11498 tablet_service.cc:1467] Processing CreateTablet for tablet 41e954768662450384ee92db7f7a4aa9 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250113 08:57:14.539415 11498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e954768662450384ee92db7f7a4aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.539921 11424 tablet_service.cc:1467] Processing CreateTablet for tablet 41e954768662450384ee92db7f7a4aa9 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250113 08:57:14.541203 11424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e954768662450384ee92db7f7a4aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.545200 11423 tablet_service.cc:1467] Processing CreateTablet for tablet 394f7fa139c04737ae3f52f53323923f (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250113 08:57:14.546528 11423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394f7fa139c04737ae3f52f53323923f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.546422 11495 tablet_service.cc:1467] Processing CreateTablet for tablet a20251bc2d314fa4b9e95d1c982716fa (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250113 08:57:14.547741 11495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20251bc2d314fa4b9e95d1c982716fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.550992 11496 tablet_service.cc:1467] Processing CreateTablet for tablet d04408dc82344bd2bce7cc2128d21e06 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250113 08:57:14.552279 11496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d04408dc82344bd2bce7cc2128d21e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.552084 11349 tablet_service.cc:1467] Processing CreateTablet for tablet 41e954768662450384ee92db7f7a4aa9 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250113 08:57:14.553369 11349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e954768662450384ee92db7f7a4aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.555651 11422 tablet_service.cc:1467] Processing CreateTablet for tablet d04408dc82344bd2bce7cc2128d21e06 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250113 08:57:14.556970 11422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d04408dc82344bd2bce7cc2128d21e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.557811 11348 tablet_service.cc:1467] Processing CreateTablet for tablet 394f7fa139c04737ae3f52f53323923f (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250113 08:57:14.558413 11421 tablet_service.cc:1467] Processing CreateTablet for tablet a20251bc2d314fa4b9e95d1c982716fa (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250113 08:57:14.559116 11348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394f7fa139c04737ae3f52f53323923f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.560632 11420 tablet_service.cc:1467] Processing CreateTablet for tablet 0f223f52b9b74b86bbde4a36265996ae (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250113 08:57:14.562414 11347 tablet_service.cc:1467] Processing CreateTablet for tablet d04408dc82344bd2bce7cc2128d21e06 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250113 08:57:14.563759 11347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d04408dc82344bd2bce7cc2128d21e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.564144 11494 tablet_service.cc:1467] Processing CreateTablet for tablet 0f223f52b9b74b86bbde4a36265996ae (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250113 08:57:14.565542 11346 tablet_service.cc:1467] Processing CreateTablet for tablet a20251bc2d314fa4b9e95d1c982716fa (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250113 08:57:14.566830 11346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20251bc2d314fa4b9e95d1c982716fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.567173 11344 tablet_service.cc:1467] Processing CreateTablet for tablet 05c4365410d14f8d9d3ab81918a2b854 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250113 08:57:14.568470 11344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05c4365410d14f8d9d3ab81918a2b854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.568233 11492 tablet_service.cc:1467] Processing CreateTablet for tablet feaff5dad2a9474e94aae9e091ffae5a (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250113 08:57:14.570500 11345 tablet_service.cc:1467] Processing CreateTablet for tablet 0f223f52b9b74b86bbde4a36265996ae (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250113 08:57:14.570922 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 9af2cb63e2c74278a497a485a1631498 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250113 08:57:14.571825 11345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f223f52b9b74b86bbde4a36265996ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.573580 11490 tablet_service.cc:1467] Processing CreateTablet for tablet a6a167a8ddd44f4b96728813a14bf98c (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250113 08:57:14.576555 11489 tablet_service.cc:1467] Processing CreateTablet for tablet 4e649934e3bf40ad8bdcf7797d0c13d7 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250113 08:57:14.576761 11343 tablet_service.cc:1467] Processing CreateTablet for tablet feaff5dad2a9474e94aae9e091ffae5a (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250113 08:57:14.566449 11493 tablet_service.cc:1467] Processing CreateTablet for tablet 05c4365410d14f8d9d3ab81918a2b854 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250113 08:57:14.578143 11343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feaff5dad2a9474e94aae9e091ffae5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.579293 11494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f223f52b9b74b86bbde4a36265996ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.581677 11489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e649934e3bf40ad8bdcf7797d0c13d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.583539 11342 tablet_service.cc:1467] Processing CreateTablet for tablet 9af2cb63e2c74278a497a485a1631498 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250113 08:57:14.584890 11342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af2cb63e2c74278a497a485a1631498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.585685 11490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a167a8ddd44f4b96728813a14bf98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.587183 11548 tablet_bootstrap.cc:492] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:14.589619 11493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05c4365410d14f8d9d3ab81918a2b854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.592378 11492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feaff5dad2a9474e94aae9e091ffae5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.595218 11548 tablet_bootstrap.cc:654] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.597586 11491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af2cb63e2c74278a497a485a1631498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.603521 11419 tablet_service.cc:1467] Processing CreateTablet for tablet 05c4365410d14f8d9d3ab81918a2b854 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250113 08:57:14.603951 11341 tablet_service.cc:1467] Processing CreateTablet for tablet a6a167a8ddd44f4b96728813a14bf98c (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250113 08:57:14.605613 11341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a167a8ddd44f4b96728813a14bf98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.607126 11418 tablet_service.cc:1467] Processing CreateTablet for tablet feaff5dad2a9474e94aae9e091ffae5a (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250113 08:57:14.609899 11420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f223f52b9b74b86bbde4a36265996ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.611321 11421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20251bc2d314fa4b9e95d1c982716fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.612668 11418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feaff5dad2a9474e94aae9e091ffae5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.614037 11419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05c4365410d14f8d9d3ab81918a2b854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.651221 11417 tablet_service.cc:1467] Processing CreateTablet for tablet 9af2cb63e2c74278a497a485a1631498 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250113 08:57:14.652982 11417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af2cb63e2c74278a497a485a1631498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.654637 11340 tablet_service.cc:1467] Processing CreateTablet for tablet 4e649934e3bf40ad8bdcf7797d0c13d7 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250113 08:57:14.655859 11340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e649934e3bf40ad8bdcf7797d0c13d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.659087 11416 tablet_service.cc:1467] Processing CreateTablet for tablet a6a167a8ddd44f4b96728813a14bf98c (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250113 08:57:14.659344 11415 tablet_service.cc:1467] Processing CreateTablet for tablet 4e649934e3bf40ad8bdcf7797d0c13d7 (DEFAULT_TABLE table=test-workload [id=21846a3dfc7f4e16a16e8beda93c91eb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250113 08:57:14.660458 11416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a167a8ddd44f4b96728813a14bf98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.665215 11415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e649934e3bf40ad8bdcf7797d0c13d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:14.692704 11550 tablet_bootstrap.cc:492] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:14.699666 11550 tablet_bootstrap.cc:654] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.706566 11550 tablet_bootstrap.cc:492] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:14.707067 11550 ts_tablet_manager.cc:1397] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20250113 08:57:14.709744 11550 raft_consensus.cc:357] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.710522 11550 raft_consensus.cc:383] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.710813 11550 raft_consensus.cc:738] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.711599 11550 consensus_queue.cc:260] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.739607 11548 tablet_bootstrap.cc:492] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:14.740268 11548 ts_tablet_manager.cc:1397] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.153s user 0.022s sys 0.071s
I20250113 08:57:14.743965 11548 raft_consensus.cc:357] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.755005 11548 raft_consensus.cc:383] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.765303 11548 raft_consensus.cc:738] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.765585 11551 tablet_bootstrap.cc:492] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:14.772302 11548 consensus_queue.cc:260] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.773813 11551 tablet_bootstrap.cc:654] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.789917 11550 ts_tablet_manager.cc:1428] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.083s user 0.027s sys 0.055s
I20250113 08:57:14.790699 11384 heartbeater.cc:502] Master 127.10.115.190:41507 was elected leader, sending a full tablet report...
I20250113 08:57:14.793805 11550 tablet_bootstrap.cc:492] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:14.803356 11548 ts_tablet_manager.cc:1428] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.063s user 0.019s sys 0.017s
I20250113 08:57:14.804141 11533 heartbeater.cc:502] Master 127.10.115.190:41507 was elected leader, sending a full tablet report...
I20250113 08:57:14.804356 11548 tablet_bootstrap.cc:492] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:14.804437 11551 tablet_bootstrap.cc:492] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:14.804975 11551 ts_tablet_manager.cc:1397] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.000s
I20250113 08:57:14.807751 11551 raft_consensus.cc:357] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.808553 11551 raft_consensus.cc:383] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.808871 11551 raft_consensus.cc:738] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.809580 11551 consensus_queue.cc:260] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.811540 11548 tablet_bootstrap.cc:654] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.815665 11550 tablet_bootstrap.cc:654] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.824126 11548 tablet_bootstrap.cc:492] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:14.824769 11548 ts_tablet_manager.cc:1397] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20250113 08:57:14.828305 11548 raft_consensus.cc:357] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:14.832151 11459 heartbeater.cc:502] Master 127.10.115.190:41507 was elected leader, sending a full tablet report...
I20250113 08:57:14.832787 11551 ts_tablet_manager.cc:1428] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.027s user 0.006s sys 0.013s
I20250113 08:57:14.833787 11551 tablet_bootstrap.cc:492] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:14.837060 11548 raft_consensus.cc:383] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.837471 11548 raft_consensus.cc:738] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.838495 11548 consensus_queue.cc:260] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:14.840270 11551 tablet_bootstrap.cc:654] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.840803 11548 ts_tablet_manager.cc:1428] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250113 08:57:14.841794 11548 tablet_bootstrap.cc:492] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:14.851085 11548 tablet_bootstrap.cc:654] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.854959 11550 tablet_bootstrap.cc:492] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:14.855484 11550 ts_tablet_manager.cc:1397] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.062s user 0.017s sys 0.026s
I20250113 08:57:14.858213 11550 raft_consensus.cc:357] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.858958 11550 raft_consensus.cc:383] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.859259 11550 raft_consensus.cc:738] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.860350 11550 consensus_queue.cc:260] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.861939 11554 raft_consensus.cc:491] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:14.862342 11554 raft_consensus.cc:513] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.862663 11550 ts_tablet_manager.cc:1428] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250113 08:57:14.863607 11550 tablet_bootstrap.cc:492] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:14.864487 11554 leader_election.cc:290] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:14.865572 11548 tablet_bootstrap.cc:492] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:14.866077 11548 ts_tablet_manager.cc:1397] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.006s
I20250113 08:57:14.869064 11548 raft_consensus.cc:357] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.869796 11548 raft_consensus.cc:383] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.870153 11548 raft_consensus.cc:738] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.870757 11548 consensus_queue.cc:260] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.873080 11548 ts_tablet_manager.cc:1428] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250113 08:57:14.874158 11548 tablet_bootstrap.cc:492] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:14.880462 11548 tablet_bootstrap.cc:654] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.891997 11548 tablet_bootstrap.cc:492] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:14.892481 11548 ts_tablet_manager.cc:1397] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20250113 08:57:14.894656 11551 tablet_bootstrap.cc:492] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:14.895215 11551 ts_tablet_manager.cc:1397] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.047s
I20250113 08:57:14.895202 11548 raft_consensus.cc:357] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.896078 11548 raft_consensus.cc:383] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.896400 11548 raft_consensus.cc:738] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.897547 11550 tablet_bootstrap.cc:654] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.897076 11548 consensus_queue.cc:260] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.899564 11548 ts_tablet_manager.cc:1428] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250113 08:57:14.900627 11548 tablet_bootstrap.cc:492] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:14.901211 11551 raft_consensus.cc:357] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.902004 11551 raft_consensus.cc:383] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.902314 11551 raft_consensus.cc:738] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.902997 11551 consensus_queue.cc:260] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.906342 11551 ts_tablet_manager.cc:1428] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20250113 08:57:14.907249 11551 tablet_bootstrap.cc:492] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:14.909132 11550 tablet_bootstrap.cc:492] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:14.909602 11550 ts_tablet_manager.cc:1397] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.008s
I20250113 08:57:14.912370 11550 raft_consensus.cc:357] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:14.913148 11550 raft_consensus.cc:383] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.913579 11550 raft_consensus.cc:738] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.913874 11551 tablet_bootstrap.cc:654] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.914373 11550 consensus_queue.cc:260] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:14.916843 11548 tablet_bootstrap.cc:654] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.921135 11550 ts_tablet_manager.cc:1428] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250113 08:57:14.922534 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:14.924265 11550 tablet_bootstrap.cc:492] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:14.925489 11551 tablet_bootstrap.cc:492] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
W20250113 08:57:14.925696 11473 leader_election.cc:343] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:14.925969 11551 ts_tablet_manager.cc:1397] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20250113 08:57:14.925725 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:14.928848 11551 raft_consensus.cc:357] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.929647 11551 raft_consensus.cc:383] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 08:57:14.927906 11473 leader_election.cc:343] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:14.929994 11551 raft_consensus.cc:738] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.930215 11473 leader_election.cc:304] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327; no voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:14.931087 11554 raft_consensus.cc:2743] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:14.930779 11551 consensus_queue.cc:260] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.933007 11550 tablet_bootstrap.cc:654] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.934146 11548 tablet_bootstrap.cc:492] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:14.934204 11551 ts_tablet_manager.cc:1428] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250113 08:57:14.934736 11548 ts_tablet_manager.cc:1397] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.001s
I20250113 08:57:14.935189 11551 tablet_bootstrap.cc:492] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:14.938200 11548 raft_consensus.cc:357] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.939170 11548 raft_consensus.cc:383] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.939518 11548 raft_consensus.cc:738] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.940485 11548 consensus_queue.cc:260] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.941783 11551 tablet_bootstrap.cc:654] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.944540 11550 tablet_bootstrap.cc:492] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:14.945014 11550 ts_tablet_manager.cc:1397] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.011s
I20250113 08:57:14.946226 11551 tablet_bootstrap.cc:492] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:14.946717 11551 ts_tablet_manager.cc:1397] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20250113 08:57:14.947029 11548 ts_tablet_manager.cc:1428] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250113 08:57:14.947727 11550 raft_consensus.cc:357] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.948480 11550 raft_consensus.cc:383] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.948771 11550 raft_consensus.cc:738] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.949138 11551 raft_consensus.cc:357] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.949755 11551 raft_consensus.cc:383] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.949993 11551 raft_consensus.cc:738] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.949482 11550 consensus_queue.cc:260] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.950546 11551 consensus_queue.cc:260] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.952224 11550 ts_tablet_manager.cc:1428] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20250113 08:57:14.953143 11550 tablet_bootstrap.cc:492] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:14.955142 11551 ts_tablet_manager.cc:1428] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20250113 08:57:14.947985 11548 tablet_bootstrap.cc:492] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:14.956058 11551 tablet_bootstrap.cc:492] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:14.960215 11550 tablet_bootstrap.cc:654] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.963806 11551 tablet_bootstrap.cc:654] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.966697 11548 tablet_bootstrap.cc:654] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:14.973469 11553 raft_consensus.cc:491] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:14.973939 11553 raft_consensus.cc:513] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.974741 11550 tablet_bootstrap.cc:492] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:14.975188 11550 ts_tablet_manager.cc:1397] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20250113 08:57:14.975740 11548 tablet_bootstrap.cc:492] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:14.976099 11548 ts_tablet_manager.cc:1397] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.003s
I20250113 08:57:14.978628 11553 leader_election.cc:290] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:14.979493 11548 raft_consensus.cc:357] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.979664 11550 raft_consensus.cc:357] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.980491 11548 raft_consensus.cc:383] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.980710 11550 raft_consensus.cc:383] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:14.980897 11548 raft_consensus.cc:738] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.981328 11550 raft_consensus.cc:738] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:14.982154 11557 raft_consensus.cc:491] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:14.981930 11548 consensus_queue.cc:260] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.982235 11550 consensus_queue.cc:260] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.982627 11557 raft_consensus.cc:513] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:14.985904 11557 leader_election.cc:290] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:14.989667 11548 ts_tablet_manager.cc:1428] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.013s user 0.007s sys 0.002s
I20250113 08:57:14.992761 11548 tablet_bootstrap.cc:492] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:14.997555 11550 ts_tablet_manager.cc:1428] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.022s user 0.005s sys 0.004s
I20250113 08:57:14.998581 11550 tablet_bootstrap.cc:492] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:15.004504 11551 tablet_bootstrap.cc:492] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:15.005056 11551 ts_tablet_manager.cc:1397] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.000s
I20250113 08:57:15.008355 11551 raft_consensus.cc:357] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.009565 11551 raft_consensus.cc:383] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.011083 11550 tablet_bootstrap.cc:654] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.011399 11551 raft_consensus.cc:738] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.015769 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.016453 11508 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 0.
I20250113 08:57:15.017764 11322 leader_election.cc:304] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:15.018616 11553 raft_consensus.cc:2798] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.018975 11553 raft_consensus.cc:491] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.019317 11553 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.022080 11548 tablet_bootstrap.cc:654] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.021646 11551 consensus_queue.cc:260] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.028537 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.029212 11434 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 0.
I20250113 08:57:15.029930 11553 raft_consensus.cc:513] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.032490 11553 leader_election.cc:290] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.033794 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:15.034369 11508 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.034799 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.035310 11550 tablet_bootstrap.cc:492] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:15.035352 11434 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.035871 11550 ts_tablet_manager.cc:1397] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20250113 08:57:15.038389 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.038648 11550 raft_consensus.cc:357] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.039080 11507 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 0.
I20250113 08:57:15.039388 11550 raft_consensus.cc:383] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.039757 11550 raft_consensus.cc:738] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.040225 11551 ts_tablet_manager.cc:1428] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20250113 08:57:15.040532 11557 raft_consensus.cc:491] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.040436 11550 consensus_queue.cc:260] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.041262 11551 tablet_bootstrap.cc:492] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:15.040984 11557 raft_consensus.cc:513] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.042860 11550 ts_tablet_manager.cc:1428] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250113 08:57:15.042925 11557 leader_election.cc:290] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.045071 11508 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 1.
I20250113 08:57:15.046645 11322 leader_election.cc:304] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:15.047995 11551 tablet_bootstrap.cc:654] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.048661 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.049971 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.050720 11550 tablet_bootstrap.cc:492] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
W20250113 08:57:15.051554 11399 leader_election.cc:343] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.050596 11358 raft_consensus.cc:2463] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 0.
I20250113 08:57:15.052881 11399 leader_election.cc:304] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.053802 11557 raft_consensus.cc:2798] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.054198 11557 raft_consensus.cc:491] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.054538 11557 raft_consensus.cc:3054] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.057358 11550 tablet_bootstrap.cc:654] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.063562 11557 raft_consensus.cc:513] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.065297 11397 leader_election.cc:304] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters: b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.066541 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.067327 11358 raft_consensus.cc:3054] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.070984 11554 raft_consensus.cc:491] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.071652 11554 raft_consensus.cc:513] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.073158 11550 tablet_bootstrap.cc:492] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:15.074144 11554 leader_election.cc:290] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
W20250113 08:57:15.075448 11305 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:15.076424 11550 ts_tablet_manager.cc:1397] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.006s
I20250113 08:57:15.076797 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.078052 11553 raft_consensus.cc:2798] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.078575 11553 raft_consensus.cc:695] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:15.079222 11550 raft_consensus.cc:357] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.079391 11553 consensus_queue.cc:237] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.080021 11550 raft_consensus.cc:383] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.079732 11433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.080329 11550 raft_consensus.cc:738] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.080696 11433 raft_consensus.cc:2463] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 0.
I20250113 08:57:15.081043 11550 consensus_queue.cc:260] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
W20250113 08:57:15.082226 11473 leader_election.cc:343] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.082891 11358 raft_consensus.cc:2463] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 1.
I20250113 08:57:15.083904 11473 leader_election.cc:304] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters: b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.084982 11554 raft_consensus.cc:2798] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.085420 11554 raft_consensus.cc:491] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.085844 11554 raft_consensus.cc:3054] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:15.089748 11460 tablet.cc:2367] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:57:15.095091 11399 leader_election.cc:304] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.097492 11583 raft_consensus.cc:2798] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.097931 11583 raft_consensus.cc:491] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.098313 11583 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.099320 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
W20250113 08:57:15.102257 11322 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:41233, user_credentials={real_user=slave}} blocked reactor thread for 55670.2us
I20250113 08:57:15.103853 11557 leader_election.cc:290] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.104606 11557 raft_consensus.cc:2798] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.105165 11557 raft_consensus.cc:695] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:15.106298 11557 consensus_queue.cc:237] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.104136 11554 raft_consensus.cc:513] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.114915 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
W20250113 08:57:15.115917 11397 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:41233, user_credentials={real_user=slave}} blocked reactor thread for 50928.9us
W20250113 08:57:15.116745 11473 leader_election.cc:343] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:15.117122 11397 leader_election.cc:343] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.113940 11583 raft_consensus.cc:513] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.119056 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327"
W20250113 08:57:15.120445 11397 leader_election.cc:343] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.111884 11434 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 1.
I20250113 08:57:15.121641 11583 leader_election.cc:290] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.124230 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:15.125319 11508 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.126029 11255 catalog_manager.cc:5526] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.130883 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.132369 11434 raft_consensus.cc:3054] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.131489 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.131605 11554 leader_election.cc:290] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
W20250113 08:57:15.142561 11399 leader_election.cc:343] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.143817 11550 ts_tablet_manager.cc:1428] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.067s user 0.029s sys 0.035s
I20250113 08:57:15.145187 11550 tablet_bootstrap.cc:492] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:15.150797 11434 raft_consensus.cc:2463] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 1.
I20250113 08:57:15.152281 11473 leader_election.cc:304] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters: b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.154954 11554 raft_consensus.cc:2798] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.155233 11508 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 1.
I20250113 08:57:15.156651 11397 leader_election.cc:304] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters: b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.158349 11581 raft_consensus.cc:2798] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.158965 11581 raft_consensus.cc:695] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:15.161149 11554 raft_consensus.cc:695] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:15.162098 11554 consensus_queue.cc:237] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.163057 11551 tablet_bootstrap.cc:492] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:15.163566 11551 ts_tablet_manager.cc:1397] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.123s user 0.039s sys 0.055s
I20250113 08:57:15.159938 11548 tablet_bootstrap.cc:492] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:15.164942 11548 ts_tablet_manager.cc:1397] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.172s user 0.016s sys 0.013s
I20250113 08:57:15.164661 11581 consensus_queue.cc:237] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.166092 11583 raft_consensus.cc:491] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.166525 11583 raft_consensus.cc:513] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.167049 11550 tablet_bootstrap.cc:654] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.167675 11548 raft_consensus.cc:357] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.170755 11583 leader_election.cc:290] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.171099 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.171609 11548 raft_consensus.cc:383] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.172482 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
W20250113 08:57:15.172847 11399 leader_election.cc:343] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.173218 11508 raft_consensus.cc:2463] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 0.
I20250113 08:57:15.173596 11548 raft_consensus.cc:738] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.174677 11397 leader_election.cc:304] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters: b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.175030 11548 consensus_queue.cc:260] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.176049 11583 raft_consensus.cc:2798] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.171267 11551 raft_consensus.cc:357] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.178025 11551 raft_consensus.cc:383] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.178390 11551 raft_consensus.cc:738] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.177276 11583 raft_consensus.cc:491] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.182662 11583 raft_consensus.cc:3054] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.179277 11551 consensus_queue.cc:260] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.189777 11591 raft_consensus.cc:491] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.191063 11591 raft_consensus.cc:513] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:15.195355 11548 ts_tablet_manager.cc:1428] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20250113 08:57:15.207378 11548 tablet_bootstrap.cc:492] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:15.207984 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.204730 11591 leader_election.cc:290] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471)
W20250113 08:57:15.212774 11473 leader_election.cc:343] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.213307 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.213793 11358 raft_consensus.cc:2463] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 0.
I20250113 08:57:15.214756 11473 leader_election.cc:304] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters: ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.215991 11592 raft_consensus.cc:2798] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.216521 11592 raft_consensus.cc:491] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.216921 11592 raft_consensus.cc:3054] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.219949 11257 catalog_manager.cc:5526] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 1 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.221336 11587 raft_consensus.cc:491] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.222048 11587 raft_consensus.cc:513] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.224109 11550 tablet_bootstrap.cc:492] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:15.226040 11592 raft_consensus.cc:513] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:15.229910 11550 ts_tablet_manager.cc:1397] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.003s
I20250113 08:57:15.229442 11582 raft_consensus.cc:491] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.231472 11582 raft_consensus.cc:513] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.232241 11592 leader_election.cc:290] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471)
I20250113 08:57:15.232352 11548 tablet_bootstrap.cc:654] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.234222 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.236574 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.237435 11358 raft_consensus.cc:3054] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.236469 11550 raft_consensus.cc:357] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.238171 11550 raft_consensus.cc:383] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.238590 11550 raft_consensus.cc:738] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.228879 11583 raft_consensus.cc:513] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
W20250113 08:57:15.240692 11473 leader_election.cc:343] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.249856 11582 leader_election.cc:290] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.250384 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
W20250113 08:57:15.252068 11324 leader_election.cc:343] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.256958 11550 consensus_queue.cc:260] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.258042 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.258674 11550 ts_tablet_manager.cc:1428] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250113 08:57:15.259141 11578 raft_consensus.cc:491] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.259526 11550 tablet_bootstrap.cc:492] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:15.259580 11578 raft_consensus.cc:513] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.263759 11578 leader_election.cc:290] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.264353 11582 raft_consensus.cc:491] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.264799 11582 raft_consensus.cc:513] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
W20250113 08:57:15.273300 11322 leader_election.cc:343] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.273737 11582 leader_election.cc:290] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.273764 11322 leader_election.cc:304] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71; no voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.277333 11578 raft_consensus.cc:2743] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:15.277967 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.277989 11433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.278875 11434 raft_consensus.cc:2388] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b793334f8c5a4a6aa947ec73fe474b71 in current term 1: Already voted for candidate 10711487b8dc43f181e340bd29bc2327 in this term.
I20250113 08:57:15.278959 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
W20250113 08:57:15.279589 11324 leader_election.cc:343] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.280417 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.281055 11587 leader_election.cc:290] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:15.282196 11322 leader_election.cc:304] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71; no voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.283258 11582 raft_consensus.cc:3054] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.286130 11550 tablet_bootstrap.cc:654] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.296765 11551 ts_tablet_manager.cc:1428] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.133s user 0.007s sys 0.014s
I20250113 08:57:15.296800 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.297981 11551 tablet_bootstrap.cc:492] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
W20250113 08:57:15.296329 11322 leader_election.cc:343] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:15.298367 11473 leader_election.cc:343] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.298547 11322 leader_election.cc:304] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71; no voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.283970 11358 raft_consensus.cc:2463] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 1.
I20250113 08:57:15.300060 11578 raft_consensus.cc:2743] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:15.306071 11473 leader_election.cc:304] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters: ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.307252 11554 raft_consensus.cc:2798] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.308890 11554 raft_consensus.cc:695] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:15.305703 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.309803 11554 consensus_queue.cc:237] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
W20250113 08:57:15.311069 11399 leader_election.cc:343] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.314342 11583 leader_election.cc:290] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.314808 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:15.315337 11507 raft_consensus.cc:3054] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.311664 11254 catalog_manager.cc:5526] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:15.317756 11582 raft_consensus.cc:2743] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:15.308351 11257 catalog_manager.cc:5526] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 1 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.321626 11548 tablet_bootstrap.cc:492] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:15.322786 11548 ts_tablet_manager.cc:1397] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.116s user 0.015s sys 0.015s
I20250113 08:57:15.324961 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.325315 11548 raft_consensus.cc:357] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.325893 11358 raft_consensus.cc:2463] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 0.
I20250113 08:57:15.325955 11507 raft_consensus.cc:2463] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 1.
I20250113 08:57:15.326100 11548 raft_consensus.cc:383] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.326695 11548 raft_consensus.cc:738] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.327400 11397 leader_election.cc:304] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters: b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.327368 11548 consensus_queue.cc:260] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.328547 11583 raft_consensus.cc:2798] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.329015 11583 raft_consensus.cc:695] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:15.329851 11583 consensus_queue.cc:237] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.332692 11548 ts_tablet_manager.cc:1428] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250113 08:57:15.333700 11548 tablet_bootstrap.cc:492] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:15.340795 11473 leader_election.cc:304] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters: ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.342464 11587 raft_consensus.cc:2798] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.342974 11587 raft_consensus.cc:491] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.343415 11587 raft_consensus.cc:3054] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.343801 11551 tablet_bootstrap.cc:654] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.349607 11550 tablet_bootstrap.cc:492] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:15.350059 11550 ts_tablet_manager.cc:1397] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.091s user 0.014s sys 0.000s
I20250113 08:57:15.352831 11550 raft_consensus.cc:357] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.353518 11550 raft_consensus.cc:383] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.353789 11550 raft_consensus.cc:738] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.354647 11550 consensus_queue.cc:260] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.356971 11550 ts_tablet_manager.cc:1428] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250113 08:57:15.352249 11257 catalog_manager.cc:5526] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 1 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.359476 11548 tablet_bootstrap.cc:654] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.361351 11550 tablet_bootstrap.cc:492] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71: Bootstrap starting.
I20250113 08:57:15.366446 11548 tablet_bootstrap.cc:492] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:15.369164 11548 ts_tablet_manager.cc:1397] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.002s
I20250113 08:57:15.370389 11254 catalog_manager.cc:5526] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.371241 11587 raft_consensus.cc:513] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.371739 11548 raft_consensus.cc:357] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.372524 11548 raft_consensus.cc:383] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.372781 11548 raft_consensus.cc:738] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.367491 11550 tablet_bootstrap.cc:654] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.368698 11551 tablet_bootstrap.cc:492] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:15.373510 11551 ts_tablet_manager.cc:1397] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.001s
I20250113 08:57:15.376576 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.376785 11551 raft_consensus.cc:357] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.378280 11551 raft_consensus.cc:383] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.378332 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.379045 11358 raft_consensus.cc:3054] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:15.379271 11473 leader_election.cc:343] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:15.379411 11548 consensus_queue.cc:260] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.378800 11551 raft_consensus.cc:738] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.381230 11551 consensus_queue.cc:260] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.384819 11548 ts_tablet_manager.cc:1428] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250113 08:57:15.372927 11587 leader_election.cc:290] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:15.387619 11551 ts_tablet_manager.cc:1428] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250113 08:57:15.388648 11548 tablet_bootstrap.cc:492] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327: Bootstrap starting.
I20250113 08:57:15.390938 11551 tablet_bootstrap.cc:492] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:15.392665 11582 raft_consensus.cc:491] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.393141 11582 raft_consensus.cc:513] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.395273 11582 leader_election.cc:290] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.396871 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.397619 11548 tablet_bootstrap.cc:654] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.398408 11551 tablet_bootstrap.cc:654] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.399183 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.399880 11434 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 0.
I20250113 08:57:15.401119 11324 leader_election.cc:304] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.397554 11507 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 0.
I20250113 08:57:15.402541 11550 tablet_bootstrap.cc:492] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71: No bootstrap required, opened a new log
I20250113 08:57:15.402973 11550 ts_tablet_manager.cc:1397] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.006s
I20250113 08:57:15.403173 11582 raft_consensus.cc:2798] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.403535 11582 raft_consensus.cc:491] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.403899 11582 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.405535 11550 raft_consensus.cc:357] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.406200 11550 raft_consensus.cc:383] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.406453 11550 raft_consensus.cc:738] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.385388 11358 raft_consensus.cc:2463] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 1.
I20250113 08:57:15.407897 11473 leader_election.cc:304] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters: ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.408119 11550 consensus_queue.cc:260] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.408730 11587 raft_consensus.cc:2798] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.409436 11587 raft_consensus.cc:695] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:15.410404 11587 consensus_queue.cc:237] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.411489 11548 tablet_bootstrap.cc:492] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327: No bootstrap required, opened a new log
I20250113 08:57:15.411933 11548 ts_tablet_manager.cc:1397] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.009s
I20250113 08:57:15.415112 11550 ts_tablet_manager.cc:1428] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250113 08:57:15.415448 11548 raft_consensus.cc:357] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.416431 11548 raft_consensus.cc:383] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.416774 11548 raft_consensus.cc:738] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.417599 11551 tablet_bootstrap.cc:492] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:15.418115 11551 ts_tablet_manager.cc:1397] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20250113 08:57:15.417678 11548 consensus_queue.cc:260] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.420644 11548 ts_tablet_manager.cc:1428] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250113 08:57:15.421154 11551 raft_consensus.cc:357] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.421873 11551 raft_consensus.cc:383] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.422160 11551 raft_consensus.cc:738] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.422837 11551 consensus_queue.cc:260] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.425033 11551 ts_tablet_manager.cc:1428] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:57:15.425913 11551 tablet_bootstrap.cc:492] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:15.429813 11582 raft_consensus.cc:513] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.431511 11551 tablet_bootstrap.cc:654] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.432507 11582 leader_election.cc:290] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.435070 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:15.435879 11507 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.433701 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.437460 11434 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.438216 11551 tablet_bootstrap.cc:492] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:15.438649 11551 ts_tablet_manager.cc:1397] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250113 08:57:15.441130 11551 raft_consensus.cc:357] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:15.441915 11551 raft_consensus.cc:383] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.442293 11551 raft_consensus.cc:738] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.443048 11551 consensus_queue.cc:260] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:15.443042 11257 catalog_manager.cc:5526] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:15.444468 11434 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 1.
I20250113 08:57:15.444886 11507 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 1.
I20250113 08:57:15.445669 11324 leader_election.cc:304] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.445981 11551 ts_tablet_manager.cc:1428] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250113 08:57:15.446765 11551 tablet_bootstrap.cc:492] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478: Bootstrap starting.
I20250113 08:57:15.446836 11582 raft_consensus.cc:2798] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.447361 11582 raft_consensus.cc:695] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:15.448110 11582 consensus_queue.cc:237] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.451673 11551 tablet_bootstrap.cc:654] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:15.455696 11587 raft_consensus.cc:491] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.456161 11587 raft_consensus.cc:513] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.457230 11551 tablet_bootstrap.cc:492] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478: No bootstrap required, opened a new log
I20250113 08:57:15.457650 11551 ts_tablet_manager.cc:1397] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20250113 08:57:15.458153 11587 leader_election.cc:290] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:15.459676 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.459914 11583 raft_consensus.cc:491] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.459813 11551 raft_consensus.cc:357] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.460489 11358 raft_consensus.cc:2388] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10711487b8dc43f181e340bd29bc2327 in current term 1: Already voted for candidate ea0cbc9eb22d4063b00e12349d64f478 in this term.
I20250113 08:57:15.460445 11583 raft_consensus.cc:513] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:15.460846 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.460489 11551 raft_consensus.cc:383] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:15.461756 11551 raft_consensus.cc:738] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Initialized, Role: FOLLOWER
I20250113 08:57:15.462469 11583 leader_election.cc:290] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233), b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471)
I20250113 08:57:15.462379 11551 consensus_queue.cc:260] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.464188 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.464584 11551 ts_tablet_manager.cc:1428] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250113 08:57:15.463979 11255 catalog_manager.cc:5526] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.465509 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.464921 11358 raft_consensus.cc:2388] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea0cbc9eb22d4063b00e12349d64f478 in current term 1: Already voted for candidate 10711487b8dc43f181e340bd29bc2327 in this term.
I20250113 08:57:15.464696 11473 leader_election.cc:304] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327; no voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.467989 11399 leader_election.cc:304] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea0cbc9eb22d4063b00e12349d64f478; no voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.468257 11587 raft_consensus.cc:3054] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.469103 11583 raft_consensus.cc:3054] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.475525 11587 raft_consensus.cc:2743] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:15.477975 11583 raft_consensus.cc:2743] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:15.483505 11587 raft_consensus.cc:491] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.483983 11587 raft_consensus.cc:513] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.485507 11587 leader_election.cc:290] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:15.486255 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.486425 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.486805 11358 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 0.
I20250113 08:57:15.487018 11434 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 0.
I20250113 08:57:15.487676 11473 leader_election.cc:304] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:15.488270 11587 raft_consensus.cc:2798] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.488531 11587 raft_consensus.cc:491] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.488794 11587 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.491454 11583 consensus_queue.cc:1035] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.493700 11587 raft_consensus.cc:513] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.495517 11587 leader_election.cc:290] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:15.496248 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.496645 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.496861 11359 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.497274 11434 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.504843 11434 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 1.
I20250113 08:57:15.505455 11359 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 1.
I20250113 08:57:15.506808 11595 consensus_queue.cc:1035] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:15.506018 11473 leader_election.cc:304] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.508003 11587 raft_consensus.cc:2798] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.508460 11587 raft_consensus.cc:695] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:15.509308 11587 consensus_queue.cc:237] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.515861 11595 raft_consensus.cc:491] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.516530 11595 raft_consensus.cc:513] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.519161 11595 leader_election.cc:290] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.520366 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.520741 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.521549 11358 raft_consensus.cc:2388] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea0cbc9eb22d4063b00e12349d64f478 in current term 1: Already voted for candidate 10711487b8dc43f181e340bd29bc2327 in this term.
I20250113 08:57:15.522440 11399 leader_election.cc:304] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea0cbc9eb22d4063b00e12349d64f478; no voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.523427 11557 raft_consensus.cc:3054] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.525576 11257 catalog_manager.cc:5526] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:15.529508 11557 raft_consensus.cc:2743] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:15.538271 11582 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.549068 11578 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.560883 11587 consensus_queue.cc:1035] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.570946 11592 consensus_queue.cc:1035] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.593783 11554 raft_consensus.cc:491] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.594403 11554 raft_consensus.cc:513] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.596669 11554 leader_election.cc:290] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:15.597784 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.597790 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71" is_pre_election: true
I20250113 08:57:15.598749 11434 raft_consensus.cc:2463] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 0.
I20250113 08:57:15.598839 11358 raft_consensus.cc:2463] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 0.
I20250113 08:57:15.599768 11473 leader_election.cc:304] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.600378 11554 raft_consensus.cc:2798] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:15.600723 11554 raft_consensus.cc:491] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:15.601015 11554 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.605702 11554 raft_consensus.cc:513] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.606644 11557 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.607388 11554 leader_election.cc:290] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:15.608166 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.608542 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.608788 11358 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.609097 11434 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.617730 11359 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.618727 11434 raft_consensus.cc:2463] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 1.
I20250113 08:57:15.619956 11473 leader_election.cc:304] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.620838 11554 raft_consensus.cc:2798] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:15.621335 11554 raft_consensus.cc:695] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:15.621454 11358 raft_consensus.cc:2463] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 1.
I20250113 08:57:15.622160 11554 consensus_queue.cc:237] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.625134 11557 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.631569 11255 catalog_manager.cc:5526] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:15.658005 11578 raft_consensus.cc:491] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:15.658430 11578 raft_consensus.cc:513] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.659888 11578 leader_election.cc:290] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.660702 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478" is_pre_election: true
I20250113 08:57:15.660964 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10711487b8dc43f181e340bd29bc2327" is_pre_election: true
I20250113 08:57:15.661738 11507 raft_consensus.cc:2388] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b793334f8c5a4a6aa947ec73fe474b71 in current term 1: Already voted for candidate ea0cbc9eb22d4063b00e12349d64f478 in this term.
I20250113 08:57:15.662528 11322 leader_election.cc:304] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71; no voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.663137 11578 raft_consensus.cc:3054] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:15.667685 11578 raft_consensus.cc:2743] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:15.692198 11554 consensus_queue.cc:1035] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.704233 11554 consensus_queue.cc:1035] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.713073 11507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54696
I20250113 08:57:15.713838 11507 raft_consensus.cc:604] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.728116 11434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54244
I20250113 08:57:15.728608 11434 raft_consensus.cc:604] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.732905 11434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54244
I20250113 08:57:15.733417 11434 raft_consensus.cc:604] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.737670 11507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54696
I20250113 08:57:15.738140 11507 raft_consensus.cc:604] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.742169 11507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54696
I20250113 08:57:15.742620 11507 raft_consensus.cc:604] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:15.754124 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:32894
I20250113 08:57:15.754621 11359 raft_consensus.cc:604] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Received request to transfer leadership to TS ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.759013 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:32894
I20250113 08:57:15.759511 11359 raft_consensus.cc:604] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Received request to transfer leadership to TS ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.764146 11507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
mode: GRACEFUL
new_leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54696
I20250113 08:57:15.764747 11507 raft_consensus.cc:604] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Received request to transfer leadership to TS ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.769114 11507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
mode: GRACEFUL
new_leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54696
I20250113 08:57:15.769632 11507 raft_consensus.cc:604] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Received request to transfer leadership to TS ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:15.773725 11434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
mode: GRACEFUL
new_leader_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:54244
I20250113 08:57:15.774216 11434 raft_consensus.cc:604] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Received request to transfer leadership to TS 10711487b8dc43f181e340bd29bc2327
I20250113 08:57:15.790030 11554 consensus_queue.cc:1035] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.796054 11592 raft_consensus.cc:988] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:15.796491 11587 raft_consensus.cc:1076] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:15.798271 11358 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32862
I20250113 08:57:15.801193 11358 raft_consensus.cc:491] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:15.801220 11592 consensus_queue.cc:1035] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:15.801771 11358 raft_consensus.cc:3054] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.807657 11358 raft_consensus.cc:513] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.813515 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.813803 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20251bc2d314fa4b9e95d1c982716fa" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:15.814258 11358 leader_election.cc:290] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.814605 11507 raft_consensus.cc:3049] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:15.814926 11507 raft_consensus.cc:738] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:15.815116 11434 raft_consensus.cc:3054] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.815760 11507 consensus_queue.cc:260] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.817102 11507 raft_consensus.cc:3054] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.822103 11434 raft_consensus.cc:2463] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:15.823328 11324 leader_election.cc:304] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.824281 11578 raft_consensus.cc:2798] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:15.824472 11507 raft_consensus.cc:2463] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:15.824827 11578 raft_consensus.cc:695] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:15.825726 11578 consensus_queue.cc:237] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.832659 11254 catalog_manager.cc:5526] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 1 to 2, leader changed from 10711487b8dc43f181e340bd29bc2327 (127.10.115.131) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.866099 11557 consensus_queue.cc:1035] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.872062 11595 raft_consensus.cc:988] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:15.872598 11583 raft_consensus.cc:1076] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:15.874275 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32878
I20250113 08:57:15.874891 11359 raft_consensus.cc:491] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:15.875269 11359 raft_consensus.cc:3054] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.879685 11595 consensus_queue.cc:1035] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.882541 11359 raft_consensus.cc:513] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.885668 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:15.886189 11434 raft_consensus.cc:3049] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:15.886425 11434 raft_consensus.cc:738] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:15.887375 11434 consensus_queue.cc:260] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.888247 11434 raft_consensus.cc:3054] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.888793 11359 leader_election.cc:290] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.890548 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04408dc82344bd2bce7cc2128d21e06" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:15.891033 11508 raft_consensus.cc:3054] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.894804 11434 raft_consensus.cc:2463] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:15.895851 11324 leader_election.cc:304] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:15.896561 11508 raft_consensus.cc:2463] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:15.896711 11578 raft_consensus.cc:2798] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:15.897320 11578 raft_consensus.cc:695] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:15.898161 11578 consensus_queue.cc:237] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.904450 11257 catalog_manager.cc:5526] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 1 to 2, leader changed from ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:15.963732 11578 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.970135 11582 raft_consensus.cc:988] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:15.970623 11598 raft_consensus.cc:1076] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Signalling peer ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:15.971897 11587 raft_consensus.cc:988] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327: : Instructing follower ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:15.972335 11587 raft_consensus.cc:1076] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Signalling peer ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:15.974665 11433 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54224
I20250113 08:57:15.975322 11433 raft_consensus.cc:491] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:15.975719 11433 raft_consensus.cc:3054] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.976367 11598 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.976779 11434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54228
I20250113 08:57:15.977347 11434 raft_consensus.cc:491] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:15.977710 11434 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.983556 11433 raft_consensus.cc:513] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.987746 11434 raft_consensus.cc:513] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.989331 11432 raft_consensus.cc:1235] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Rejecting Update request from peer b793334f8c5a4a6aa947ec73fe474b71 for earlier term 1. Current term is 2. Ops: []
I20250113 08:57:15.990124 11433 leader_election.cc:290] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.989339 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:15.991555 11359 raft_consensus.cc:3054] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.990890 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a167a8ddd44f4b96728813a14bf98c" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:15.994953 11507 raft_consensus.cc:3049] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:15.995296 11507 raft_consensus.cc:738] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:15.996443 11507 consensus_queue.cc:260] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:15.997406 11578 consensus_queue.cc:1046] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:15.997839 11507 raft_consensus.cc:3054] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:15.998404 11598 raft_consensus.cc:3049] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:15.998764 11598 raft_consensus.cc:738] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:15.999575 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:16.000139 11434 leader_election.cc:290] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:15.999558 11598 consensus_queue.cc:260] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.001564 11598 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.001444 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:16.002135 11508 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.003962 11359 raft_consensus.cc:2463] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 2.
I20250113 08:57:16.005183 11399 leader_election.cc:304] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:16.006023 11595 raft_consensus.cc:2798] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.006529 11595 raft_consensus.cc:695] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:16.007411 11595 consensus_queue.cc:237] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.010851 11507 raft_consensus.cc:2463] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 2.
I20250113 08:57:16.012755 11508 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 2.
I20250113 08:57:16.014431 11397 leader_election.cc:304] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:16.015220 11358 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 2.
I20250113 08:57:16.016184 11595 raft_consensus.cc:2798] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.016758 11595 raft_consensus.cc:695] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:16.017717 11595 consensus_queue.cc:237] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.019234 11254 catalog_manager.cc:5526] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 1 to 2, leader changed from 10711487b8dc43f181e340bd29bc2327 (127.10.115.131) to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 2 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:16.020257 11592 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:16.027041 11587 raft_consensus.cc:988] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:16.027468 11587 raft_consensus.cc:1076] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:16.029398 11358 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32862
I20250113 08:57:16.029975 11358 raft_consensus.cc:491] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:16.030352 11358 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.035266 11592 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:16.036152 11254 catalog_manager.cc:5526] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 1 to 2, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 2 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:16.043185 11358 raft_consensus.cc:513] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.044123 11359 raft_consensus.cc:1235] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Rejecting Update request from peer 10711487b8dc43f181e340bd29bc2327 for earlier term 1. Current term is 2. Ops: []
I20250113 08:57:16.045387 11358 leader_election.cc:290] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:16.045864 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:16.046188 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:16.046473 11434 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.046780 11508 raft_consensus.cc:3049] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:16.047145 11508 raft_consensus.cc:738] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:16.047868 11508 consensus_queue.cc:260] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.048677 11508 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.051499 11434 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:16.052455 11324 leader_election.cc:304] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:16.053097 11598 raft_consensus.cc:2798] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.053522 11598 raft_consensus.cc:695] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:16.054100 11508 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:16.054339 11598 consensus_queue.cc:237] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.060516 11254 catalog_manager.cc:5526] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 1 to 2, leader changed from 10711487b8dc43f181e340bd29bc2327 (127.10.115.131) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:16.082371 11578 raft_consensus.cc:988] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:16.082840 11598 raft_consensus.cc:1076] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Signalling peer ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:16.084295 11434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54228
I20250113 08:57:16.084887 11434 raft_consensus.cc:491] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:16.085309 11434 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.092542 11434 raft_consensus.cc:513] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.094064 11434 leader_election.cc:290] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:16.095024 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:16.095119 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:16.095557 11359 raft_consensus.cc:3049] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:16.095654 11508 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.095827 11359 raft_consensus.cc:738] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:16.096557 11359 consensus_queue.cc:260] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.097556 11359 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.100584 11508 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 2.
I20250113 08:57:16.101841 11397 leader_election.cc:304] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:16.102615 11595 raft_consensus.cc:2798] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.102661 11359 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 2.
I20250113 08:57:16.103214 11595 raft_consensus.cc:695] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:16.104024 11592 consensus_queue.cc:1035] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:16.104106 11595 consensus_queue.cc:237] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.110769 11557 raft_consensus.cc:988] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478: : Instructing follower 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:16.111212 11557 raft_consensus.cc:1076] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Signalling peer 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:16.113011 11508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:54680
I20250113 08:57:16.113602 11508 raft_consensus.cc:491] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:16.113965 11508 raft_consensus.cc:3054] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.117718 11257 catalog_manager.cc:5526] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 1 to 2, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 2 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:16.119488 11587 raft_consensus.cc:988] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327: : Instructing follower ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:16.119942 11587 raft_consensus.cc:1076] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Signalling peer ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:16.120874 11508 raft_consensus.cc:513] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.122247 11434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54224
I20250113 08:57:16.122802 11434 raft_consensus.cc:491] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:16.123157 11434 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.123525 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:16.124102 11358 raft_consensus.cc:3054] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.124755 11433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e649934e3bf40ad8bdcf7797d0c13d7" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:16.125368 11433 raft_consensus.cc:3049] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:16.125672 11433 raft_consensus.cc:738] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:16.126431 11433 consensus_queue.cc:260] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.127578 11433 raft_consensus.cc:3054] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.129433 11639 consensus_queue.cc:1035] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:16.130249 11434 raft_consensus.cc:513] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.133734 11508 leader_election.cc:290] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 2 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:16.133620 11357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:16.134301 11358 raft_consensus.cc:2463] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 2.
I20250113 08:57:16.135741 11473 leader_election.cc:304] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:16.136530 11433 raft_consensus.cc:2463] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 2.
I20250113 08:57:16.137086 11592 raft_consensus.cc:2798] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.137575 11592 raft_consensus.cc:695] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 2 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:16.137900 11432 raft_consensus.cc:1235] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Rejecting Update request from peer 10711487b8dc43f181e340bd29bc2327 for earlier term 1. Current term is 2. Ops: []
I20250113 08:57:16.138449 11592 consensus_queue.cc:237] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.139420 11587 consensus_queue.cc:1046] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:16.140412 11587 raft_consensus.cc:3049] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:16.140947 11587 raft_consensus.cc:738] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:16.141034 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:16.133644 11434 leader_election.cc:290] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:16.142323 11587 consensus_queue.cc:260] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.142923 11557 raft_consensus.cc:988] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:16.143381 11557 raft_consensus.cc:1076] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:16.143544 11587 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.145174 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32878
I20250113 08:57:16.145829 11359 raft_consensus.cc:491] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:16.146198 11359 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.153160 11359 raft_consensus.cc:513] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.155243 11359 leader_election.cc:290] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:16.156183 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:16.156805 11432 raft_consensus.cc:3049] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:16.157145 11432 raft_consensus.cc:738] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:16.157003 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:16.157914 11508 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.157298 11257 catalog_manager.cc:5526] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 1 to 2, leader changed from ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130) to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:16.158118 11432 consensus_queue.cc:260] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.159430 11432 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.165163 11508 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:16.165795 11507 raft_consensus.cc:2463] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 2.
I20250113 08:57:16.166757 11322 leader_election.cc:304] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:16.167450 11397 leader_election.cc:304] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters: b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:16.167625 11598 raft_consensus.cc:2798] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.166743 11432 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:16.168393 11595 raft_consensus.cc:2798] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.168835 11598 raft_consensus.cc:695] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:16.168943 11595 raft_consensus.cc:695] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:16.169661 11598 consensus_queue.cc:237] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.169806 11595 consensus_queue.cc:237] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:16.176482 11257 catalog_manager.cc:5526] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 1 to 2, leader changed from ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:16.176918 11254 catalog_manager.cc:5526] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 1 to 2, leader changed from 10711487b8dc43f181e340bd29bc2327 (127.10.115.131) to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 2 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:16.184689 11587 raft_consensus.cc:988] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:16.185175 11639 raft_consensus.cc:1076] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:16.186622 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32862
I20250113 08:57:16.187052 11359 raft_consensus.cc:491] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:16.187316 11359 raft_consensus.cc:3054] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.191884 11359 raft_consensus.cc:513] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:16.193446 11359 leader_election.cc:290] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:16.194342 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:16.194465 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e954768662450384ee92db7f7a4aa9" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:16.195015 11432 raft_consensus.cc:3054] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.195075 11507 raft_consensus.cc:3049] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:16.195513 11507 raft_consensus.cc:738] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:16.196314 11507 consensus_queue.cc:260] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:16.197616 11507 raft_consensus.cc:3054] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.202044 11432 raft_consensus.cc:2463] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:16.203212 11324 leader_election.cc:304] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:16.204061 11598 raft_consensus.cc:2798] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:16.204396 11507 raft_consensus.cc:2463] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 2.
I20250113 08:57:16.204576 11598 raft_consensus.cc:695] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:16.205523 11598 consensus_queue.cc:237] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } }
I20250113 08:57:16.211881 11254 catalog_manager.cc:5526] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 1 to 2, leader changed from 10711487b8dc43f181e340bd29bc2327 (127.10.115.131) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:16.254570 11432 raft_consensus.cc:1270] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.255685 11598 consensus_queue.cc:1035] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.262609 11507 raft_consensus.cc:1270] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.264187 11582 consensus_queue.cc:1035] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.446317 11432 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.447540 11582 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.454893 11507 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.459498 11582 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.462838 11506 raft_consensus.cc:1270] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.462937 11507 raft_consensus.cc:1270] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.464463 11582 consensus_queue.cc:1035] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.465044 11595 consensus_queue.cc:1035] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.473699 11359 raft_consensus.cc:1270] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.476480 11595 consensus_queue.cc:1035] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:57:16.477305 11432 raft_consensus.cc:1270] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.485445 11582 consensus_queue.cc:1035] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:57:16.544641 11508 raft_consensus.cc:1270] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.546098 11595 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.553304 11359 raft_consensus.cc:1270] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.554770 11642 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:57:16.577350 11507 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.577350 11508 raft_consensus.cc:1270] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.578843 11642 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:57:16.578843 11582 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:57:16.587836 11432 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.589886 11598 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.591951 11359 raft_consensus.cc:1270] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.594062 11595 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.646723 11508 raft_consensus.cc:1270] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.647846 11642 consensus_queue.cc:1035] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.654882 11359 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:16.660763 11359 raft_consensus.cc:1270] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.661870 11583 consensus_queue.cc:1035] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.693749 11359 raft_consensus.cc:1270] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.694831 11639 consensus_queue.cc:1035] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.702773 11432 raft_consensus.cc:1270] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.704180 11645 consensus_queue.cc:1035] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.800174 11432 raft_consensus.cc:1270] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.801380 11582 consensus_queue.cc:1035] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:16.809369 11508 raft_consensus.cc:1270] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:16.810904 11598 consensus_queue.cc:1035] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:57:18.803328 10702 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20250113 08:57:18.803964 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:18.823722 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:32896
I20250113 08:57:18.824249 11359 raft_consensus.cc:604] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Received request to transfer leadership to TS 10711487b8dc43f181e340bd29bc2327
I20250113 08:57:18.826330 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:32896
I20250113 08:57:18.826756 11359 raft_consensus.cc:604] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Received request to transfer leadership to TS 10711487b8dc43f181e340bd29bc2327
I20250113 08:57:18.827591 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250113 08:57:18.827970 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:18.858278 11652 raft_consensus.cc:988] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:18.858688 11635 raft_consensus.cc:1076] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Signalling peer 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:18.860085 11508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:54670
I20250113 08:57:18.860569 11508 raft_consensus.cc:491] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:18.860859 11508 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:18.865553 11508 raft_consensus.cc:513] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:18.867040 11508 leader_election.cc:290] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 3 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:18.867815 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:18.868000 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:18.868392 11359 raft_consensus.cc:3049] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:18.868584 11432 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:18.868690 11359 raft_consensus.cc:738] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:18.869462 11359 consensus_queue.cc:260] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:18.870405 11359 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:18.873410 11432 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 3.
I20250113 08:57:18.874588 11473 leader_election.cc:304] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:18.875185 11359 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 3.
I20250113 08:57:18.875342 11651 raft_consensus.cc:2798] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:18.876602 11651 raft_consensus.cc:695] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:18.877529 11651 consensus_queue.cc:237] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:18.883877 11256 catalog_manager.cc:5526] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 2 to 3, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 3 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:19.275646 11652 raft_consensus.cc:988] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:19.276070 11635 raft_consensus.cc:1076] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Signalling peer 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:19.277493 11507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:54670
I20250113 08:57:19.277956 11507 raft_consensus.cc:491] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:19.278353 11507 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:19.282768 11507 raft_consensus.cc:513] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:19.284132 11507 leader_election.cc:290] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 3 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:19.285003 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:19.285202 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:19.285557 11359 raft_consensus.cc:3049] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:19.285741 11432 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:19.285913 11359 raft_consensus.cc:738] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:19.286605 11359 consensus_queue.cc:260] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:19.287490 11359 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:19.290350 11432 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 3.
I20250113 08:57:19.291218 11473 leader_election.cc:304] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:19.291803 11651 raft_consensus.cc:2798] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:19.291993 11359 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 3.
I20250113 08:57:19.292150 11651 raft_consensus.cc:695] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:19.293208 11651 consensus_queue.cc:237] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:19.299422 11256 catalog_manager.cc:5526] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 2 to 3, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 3 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:19.329145 11359 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:19.330273 11651 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:19.340332 11432 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:19.342298 11664 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:57:19.736836 11359 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:19.738288 11651 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:19.753397 11432 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:19.754822 11655 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:19.828768 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:19.834072 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:19.834506 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:20.835335 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:20.841121 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:20.841673 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:21.842502 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:21.847901 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:21.848326 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:23.257479 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:23.262913 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:23.263298 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250113 08:57:23.909745 11455 debug-util.cc:398] Leaking SignalData structure 0x7b080010b9a0 after lost signal to thread 10705
W20250113 08:57:23.910631 11455 debug-util.cc:398] Leaking SignalData structure 0x7b0800107080 after lost signal to thread 11291
W20250113 08:57:23.911453 11455 debug-util.cc:398] Leaking SignalData structure 0x7b08000f9e80 after lost signal to thread 11383
W20250113 08:57:23.912249 11455 debug-util.cc:398] Leaking SignalData structure 0x7b08000dd600 after lost signal to thread 11458
W20250113 08:57:23.912976 11455 debug-util.cc:398] Leaking SignalData structure 0x7b0800146540 after lost signal to thread 11532
I20250113 08:57:24.328658 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:24.343050 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:24.344233 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250113 08:57:25.304728 11380 debug-util.cc:398] Leaking SignalData structure 0x7b08001a76e0 after lost signal to thread 10705
W20250113 08:57:25.305528 11380 debug-util.cc:398] Leaking SignalData structure 0x7b08001e8660 after lost signal to thread 11291
W20250113 08:57:25.306388 11380 debug-util.cc:398] Leaking SignalData structure 0x7b08001df1a0 after lost signal to thread 11383
W20250113 08:57:25.307096 11380 debug-util.cc:398] Leaking SignalData structure 0x7b0800114520 after lost signal to thread 11458
W20250113 08:57:25.307833 11380 debug-util.cc:398] Leaking SignalData structure 0x7b08000adfe0 after lost signal to thread 11532
I20250113 08:57:25.346361 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:25.353722 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:25.354203 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:26.355522 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:26.365763 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:26.366438 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:27.368382 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:27.373934 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:27.374408 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:28.375113 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:28.380424 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:28.380831 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:29.416818 10702 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20250113 08:57:29.425365 11508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54696
I20250113 08:57:29.426447 11508 raft_consensus.cc:604] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:29.437803 11508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54696
I20250113 08:57:29.438285 11508 raft_consensus.cc:604] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:29.443819 11434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54244
I20250113 08:57:29.444505 11434 raft_consensus.cc:604] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:29.450417 11434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54244
I20250113 08:57:29.450876 11434 raft_consensus.cc:604] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:29.455677 11434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
mode: GRACEFUL
new_leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:54244
I20250113 08:57:29.456116 11434 raft_consensus.cc:604] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Received request to transfer leadership to TS b793334f8c5a4a6aa947ec73fe474b71
I20250113 08:57:29.468997 11686 raft_consensus.cc:988] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.469951 11691 raft_consensus.cc:1076] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.473193 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32878
I20250113 08:57:29.473714 11359 raft_consensus.cc:491] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:29.474231 11359 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.485508 11359 raft_consensus.cc:513] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.488582 11359 leader_election.cc:290] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 3 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:29.490577 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:29.490504 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:29.491755 11434 raft_consensus.cc:3049] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:29.491812 11508 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.492100 11434 raft_consensus.cc:738] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:29.493520 11434 consensus_queue.cc:260] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.495057 11434 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.501538 11508 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 3.
I20250113 08:57:29.503435 11322 leader_election.cc:304] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:29.504341 11434 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 3.
I20250113 08:57:29.505097 11685 raft_consensus.cc:2798] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:29.506572 11685 raft_consensus.cc:695] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:29.508029 11685 consensus_queue.cc:237] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.520807 11256 catalog_manager.cc:5526] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 2 to 3, leader changed from ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:29.622399 11664 raft_consensus.cc:988] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.623406 11698 raft_consensus.cc:1076] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.625310 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32862
I20250113 08:57:29.625730 11359 raft_consensus.cc:491] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:29.625948 11359 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:29.631806 11359 raft_consensus.cc:513] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.633090 11359 leader_election.cc:290] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 4 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:29.634321 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:29.634326 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:29.634827 11508 raft_consensus.cc:3049] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:29.634914 11434 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:29.635071 11508 raft_consensus.cc:738] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:29.635777 11508 consensus_queue.cc:260] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.636822 11508 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:29.639526 11434 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 4.
I20250113 08:57:29.641206 11324 leader_election.cc:304] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:29.642257 11508 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 4.
I20250113 08:57:29.642467 11685 raft_consensus.cc:2798] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:29.643066 11685 raft_consensus.cc:695] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:29.644217 11685 consensus_queue.cc:237] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.654392 11256 catalog_manager.cc:5526] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 3 to 4, leader changed from 10711487b8dc43f181e340bd29bc2327 (127.10.115.131) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 4 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:29.709357 11686 raft_consensus.cc:988] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.709883 11691 raft_consensus.cc:1076] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.711743 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32878
I20250113 08:57:29.712152 11359 raft_consensus.cc:491] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:29.712431 11359 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.716763 11359 raft_consensus.cc:513] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.717025 11686 raft_consensus.cc:988] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.717798 11691 raft_consensus.cc:1076] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.718528 11359 leader_election.cc:290] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 3 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:29.719547 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:29.719692 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af2cb63e2c74278a497a485a1631498" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:29.720014 11434 raft_consensus.cc:3049] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:29.720353 11434 raft_consensus.cc:738] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:29.720163 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32878
I20250113 08:57:29.720242 11508 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.720996 11359 raft_consensus.cc:491] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:29.721405 11359 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.721290 11434 consensus_queue.cc:260] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.722293 11434 raft_consensus.cc:3054] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.727779 11359 raft_consensus.cc:513] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.728646 11508 raft_consensus.cc:2463] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 3.
I20250113 08:57:29.728905 11664 raft_consensus.cc:988] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: : Instructing follower b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.729547 11698 raft_consensus.cc:1076] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Signalling peer b793334f8c5a4a6aa947ec73fe474b71 to start an election
I20250113 08:57:29.729681 11359 leader_election.cc:290] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 3 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:29.730715 11322 leader_election.cc:304] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:29.731617 11434 raft_consensus.cc:2463] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 3.
I20250113 08:57:29.731676 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:29.732179 11685 raft_consensus.cc:2798] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:29.732663 11432 raft_consensus.cc:3049] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:57:29.733162 11432 raft_consensus.cc:738] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:29.732992 11685 raft_consensus.cc:695] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:29.733978 11359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
from {username='slave'} at 127.0.0.1:32862
I20250113 08:57:29.734615 11359 raft_consensus.cc:491] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:29.734473 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:29.734956 11359 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:29.735105 11507 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.734851 11432 consensus_queue.cc:260] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.735249 11685 consensus_queue.cc:237] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.737105 11432 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:29.742630 11359 raft_consensus.cc:513] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.744611 11507 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 3.
I20250113 08:57:29.745725 11359 leader_election.cc:290] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 4 election: Requested vote from peers ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:29.745904 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:29.745826 11322 leader_election.cc:304] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, b793334f8c5a4a6aa947ec73fe474b71; no voters:
I20250113 08:57:29.746631 11434 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:29.747797 11692 raft_consensus.cc:2798] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:57:29.749151 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "b793334f8c5a4a6aa947ec73fe474b71" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:29.749823 11508 raft_consensus.cc:3049] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:29.750192 11508 raft_consensus.cc:738] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:29.751125 11432 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 3.
I20250113 08:57:29.751283 11508 consensus_queue.cc:260] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.752970 11508 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:29.754138 11434 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 4.
I20250113 08:57:29.755614 11324 leader_election.cc:304] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b793334f8c5a4a6aa947ec73fe474b71, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:29.756853 11685 raft_consensus.cc:2798] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:29.757635 11685 raft_consensus.cc:695] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:29.759003 11685 consensus_queue.cc:237] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.760394 11508 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b793334f8c5a4a6aa947ec73fe474b71 in term 4.
I20250113 08:57:29.760979 11692 raft_consensus.cc:695] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Becoming Leader. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:29.762897 11692 consensus_queue.cc:237] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:29.763533 11256 catalog_manager.cc:5526] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 2 to 3, leader changed from ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:29.781333 11256 catalog_manager.cc:5526] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 3 to 4, leader changed from 10711487b8dc43f181e340bd29bc2327 (127.10.115.131) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 4 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:29.782480 11256 catalog_manager.cc:5526] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 reported cstate change: term changed from 2 to 3, leader changed from ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130) to b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "b793334f8c5a4a6aa947ec73fe474b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:30.056535 11432 raft_consensus.cc:1270] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:30.057979 11692 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:30.072579 11508 raft_consensus.cc:1270] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:30.074113 11685 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:57:30.119701 11432 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:30.121340 11685 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:30.133801 11508 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:30.136523 11701 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:57:30.155678 11508 raft_consensus.cc:1270] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:30.156818 11701 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:30.170459 11432 raft_consensus.cc:1270] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:30.172959 11685 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:57:30.183212 11432 raft_consensus.cc:1270] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:30.184432 11685 consensus_queue.cc:1035] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:30.194075 11508 raft_consensus.cc:1270] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:57:30.198369 11701 consensus_queue.cc:1035] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:57:30.227721 11508 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:30.228868 11701 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:30.239040 11432 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Refusing update from remote peer b793334f8c5a4a6aa947ec73fe474b71: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:30.240304 11710 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:32.501669 10702 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20250113 08:57:32.502569 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:32.509476 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:32896
I20250113 08:57:32.510125 11359 raft_consensus.cc:604] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Received request to transfer leadership to TS 10711487b8dc43f181e340bd29bc2327
I20250113 08:57:32.512459 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:32896
I20250113 08:57:32.512893 11359 raft_consensus.cc:604] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Received request to transfer leadership to TS 10711487b8dc43f181e340bd29bc2327
I20250113 08:57:32.514844 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:32896
I20250113 08:57:32.515228 11359 raft_consensus.cc:604] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Received request to transfer leadership to TS ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:32.517174 11359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a"
dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
mode: GRACEFUL
new_leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:32896
I20250113 08:57:32.517647 11359 raft_consensus.cc:604] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Received request to transfer leadership to TS ea0cbc9eb22d4063b00e12349d64f478
I20250113 08:57:32.518491 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250113 08:57:32.518926 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:32.603061 11701 raft_consensus.cc:988] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:32.603520 11701 raft_consensus.cc:1076] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Signalling peer 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:32.605010 11508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:54670
I20250113 08:57:32.605521 11508 raft_consensus.cc:491] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:32.605840 11508 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:32.611667 11508 raft_consensus.cc:513] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.614248 11508 leader_election.cc:290] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 5 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:32.614905 11701 raft_consensus.cc:988] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:32.615077 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:32.615347 11434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223f52b9b74b86bbde4a36265996ae" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:32.616173 11719 raft_consensus.cc:1076] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Signalling peer ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:32.616443 11359 raft_consensus.cc:3049] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:57:32.616540 11434 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:32.616806 11359 raft_consensus.cc:738] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:32.618168 11432 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54228
I20250113 08:57:32.617774 11359 consensus_queue.cc:260] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.618743 11432 raft_consensus.cc:491] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:32.619123 11432 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:32.619239 11359 raft_consensus.cc:3054] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:32.622507 11434 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 5.
I20250113 08:57:32.623937 11473 leader_election.cc:304] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:32.624807 11698 raft_consensus.cc:2798] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:57:32.626462 11359 raft_consensus.cc:2463] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 5.
I20250113 08:57:32.626514 11432 raft_consensus.cc:513] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.628137 11719 raft_consensus.cc:988] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:32.628695 11701 raft_consensus.cc:1076] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Signalling peer 10711487b8dc43f181e340bd29bc2327 to start an election
I20250113 08:57:32.628163 11698 raft_consensus.cc:695] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 5 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:32.629750 11432 leader_election.cc:290] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 5 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:32.630342 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:32.630038 11698 consensus_queue.cc:237] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.630877 11359 raft_consensus.cc:3049] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:57:32.631168 11359 raft_consensus.cc:738] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:32.631947 11359 consensus_queue.cc:260] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.633338 11359 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:32.634873 11507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854"
dest_uuid: "10711487b8dc43f181e340bd29bc2327"
from {username='slave'} at 127.0.0.1:54670
I20250113 08:57:32.634935 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394f7fa139c04737ae3f52f53323923f" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:32.635515 11507 raft_consensus.cc:491] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:32.635684 11508 raft_consensus.cc:3054] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:57:32.635897 11507 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:32.640173 11508 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 5.
I20250113 08:57:32.641399 11397 leader_election.cc:304] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:32.642386 11691 raft_consensus.cc:2798] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:57:32.642876 11507 raft_consensus.cc:513] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.643540 11691 raft_consensus.cc:695] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 5 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:32.644014 11255 catalog_manager.cc:5526] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 4 to 5, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 5 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:32.644548 11691 consensus_queue.cc:237] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.645304 11507 leader_election.cc:290] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 4 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101)
I20250113 08:57:32.646416 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
I20250113 08:57:32.647141 11432 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:32.647704 11358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c4365410d14f8d9d3ab81918a2b854" candidate_uuid: "10711487b8dc43f181e340bd29bc2327" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:32.648444 11358 raft_consensus.cc:3049] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:32.648824 11358 raft_consensus.cc:738] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:32.650400 11359 raft_consensus.cc:2463] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 5.
I20250113 08:57:32.649925 11358 consensus_queue.cc:260] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.651724 11358 raft_consensus.cc:3054] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:32.654662 11432 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 4.
I20250113 08:57:32.656378 11473 leader_election.cc:304] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:32.657124 11698 raft_consensus.cc:2798] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:32.657722 11698 raft_consensus.cc:695] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 4 LEADER]: Becoming Leader. State: Replica: 10711487b8dc43f181e340bd29bc2327, State: Running, Role: LEADER
I20250113 08:57:32.658686 11698 consensus_queue.cc:237] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.661819 11358 raft_consensus.cc:2463] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10711487b8dc43f181e340bd29bc2327 in term 4.
I20250113 08:57:32.663280 11256 catalog_manager.cc:5526] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 4 to 5, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 5 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:32.677951 11256 catalog_manager.cc:5526] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 reported cstate change: term changed from 3 to 4, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to 10711487b8dc43f181e340bd29bc2327 (127.10.115.131). New cstate: current_term: 4 leader_uuid: "10711487b8dc43f181e340bd29bc2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:32.692895 11719 raft_consensus.cc:988] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: : Instructing follower ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:32.693325 11701 raft_consensus.cc:1076] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Signalling peer ea0cbc9eb22d4063b00e12349d64f478 to start an election
I20250113 08:57:32.694598 11432 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a"
dest_uuid: "ea0cbc9eb22d4063b00e12349d64f478"
from {username='slave'} at 127.0.0.1:54228
I20250113 08:57:32.694998 11432 raft_consensus.cc:491] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:57:32.695228 11432 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:32.700855 11432 raft_consensus.cc:513] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.702386 11432 leader_election.cc:290] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 4 election: Requested vote from peers b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471), 10711487b8dc43f181e340bd29bc2327 (127.10.115.131:41233)
I20250113 08:57:32.703019 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b793334f8c5a4a6aa947ec73fe474b71"
I20250113 08:57:32.703349 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feaff5dad2a9474e94aae9e091ffae5a" candidate_uuid: "ea0cbc9eb22d4063b00e12349d64f478" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "10711487b8dc43f181e340bd29bc2327"
I20250113 08:57:32.703573 11359 raft_consensus.cc:3049] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:57:32.703862 11359 raft_consensus.cc:738] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b793334f8c5a4a6aa947ec73fe474b71, State: Running, Role: LEADER
I20250113 08:57:32.703948 11507 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:32.704640 11359 consensus_queue.cc:260] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.705909 11359 raft_consensus.cc:3054] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:57:32.710208 11507 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 4.
I20250113 08:57:32.711179 11397 leader_election.cc:304] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10711487b8dc43f181e340bd29bc2327, ea0cbc9eb22d4063b00e12349d64f478; no voters:
I20250113 08:57:32.712080 11359 raft_consensus.cc:2463] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea0cbc9eb22d4063b00e12349d64f478 in term 4.
I20250113 08:57:32.712111 11722 raft_consensus.cc:2798] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:57:32.712656 11722 raft_consensus.cc:695] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 4 LEADER]: Becoming Leader. State: Replica: ea0cbc9eb22d4063b00e12349d64f478, State: Running, Role: LEADER
I20250113 08:57:32.713474 11722 consensus_queue.cc:237] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } }
I20250113 08:57:32.719527 11256 catalog_manager.cc:5526] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 reported cstate change: term changed from 3 to 4, leader changed from b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129) to ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130). New cstate: current_term: 4 leader_uuid: "ea0cbc9eb22d4063b00e12349d64f478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:33.062245 11359 raft_consensus.cc:1270] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:33.063793 11698 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:33.075820 11432 raft_consensus.cc:1270] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:33.077064 11698 consensus_queue.cc:1035] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:33.124212 11432 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 5 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250113 08:57:33.125424 11698 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cbc9eb22d4063b00e12349d64f478" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:57:33.134197 11359 raft_consensus.cc:1270] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Refusing update from remote peer 10711487b8dc43f181e340bd29bc2327: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250113 08:57:33.135903 11698 consensus_queue.cc:1035] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:57:33.202162 11359 raft_consensus.cc:1270] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:33.203289 11722 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:33.211620 11507 raft_consensus.cc:1270] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:57:33.213156 11508 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 5 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250113 08:57:33.213712 11725 consensus_queue.cc:1035] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:57:33.214409 11722 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10711487b8dc43f181e340bd29bc2327" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:57:33.228085 11359 raft_consensus.cc:1270] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Refusing update from remote peer ea0cbc9eb22d4063b00e12349d64f478: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250113 08:57:33.230381 11741 consensus_queue.cc:1035] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b793334f8c5a4a6aa947ec73fe474b71" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:57:33.519805 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:33.525115 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:33.525561 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:34.526271 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:34.531306 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:34.531697 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:35.532509 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:35.538960 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:35.539366 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:36.650717 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:36.660441 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:36.661190 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250113 08:57:37.560427 11529 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae4a0 after lost signal to thread 10705
W20250113 08:57:37.561337 11529 debug-util.cc:398] Leaking SignalData structure 0x7b08000aeb80 after lost signal to thread 11291
W20250113 08:57:37.562150 11529 debug-util.cc:398] Leaking SignalData structure 0x7b08001bba40 after lost signal to thread 11383
W20250113 08:57:37.562848 11529 debug-util.cc:398] Leaking SignalData structure 0x7b0800202680 after lost signal to thread 11458
W20250113 08:57:37.563535 11529 debug-util.cc:398] Leaking SignalData structure 0x7b08001a14a0 after lost signal to thread 11532
I20250113 08:57:37.662209 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:37.667819 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:37.668299 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:38.669166 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:38.675426 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:38.675850 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:39.676653 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:39.682109 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:39.682564 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:40.683458 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:40.689191 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:40.689685 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:41.690436 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:57:41.695971 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250113 08:57:41.696413 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:57:42.724442 10702 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20250113 08:57:42.725073 10702 tablet_server.cc:178] TabletServer@127.10.115.129:0 shutting down...
W20250113 08:57:42.735637 11399 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20250113 08:57:42.745932 11399 consensus_peers.cc:487] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 -> Peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Couldn't send request to peer b793334f8c5a4a6aa947ec73fe474b71. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:42.758323 10702 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 08:57:42.759292 10702 tablet_replica.cc:331] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.760316 10702 raft_consensus.cc:2238] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Raft consensus shutting down.
I20250113 08:57:42.761679 10702 raft_consensus.cc:2267] T d04408dc82344bd2bce7cc2128d21e06 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250113 08:57:42.761747 11399 consensus_peers.cc:487] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 -> Peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Couldn't send request to peer b793334f8c5a4a6aa947ec73fe474b71. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:42.763881 10702 tablet_replica.cc:331] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.764518 10702 raft_consensus.cc:2238] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.765015 10702 raft_consensus.cc:2267] T 394f7fa139c04737ae3f52f53323923f P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.766999 10702 tablet_replica.cc:331] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.767467 10702 raft_consensus.cc:2238] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.767846 10702 raft_consensus.cc:2267] T 0f223f52b9b74b86bbde4a36265996ae P b793334f8c5a4a6aa947ec73fe474b71 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.769603 10702 tablet_replica.cc:331] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.770180 10702 raft_consensus.cc:2238] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.770601 10702 raft_consensus.cc:2267] T feaff5dad2a9474e94aae9e091ffae5a P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.772395 10702 tablet_replica.cc:331] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.772913 10702 raft_consensus.cc:2238] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.773375 10702 raft_consensus.cc:2267] T 4e649934e3bf40ad8bdcf7797d0c13d7 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.775060 10702 tablet_replica.cc:331] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.775552 10702 raft_consensus.cc:2238] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.775978 10702 raft_consensus.cc:2267] T 05c4365410d14f8d9d3ab81918a2b854 P b793334f8c5a4a6aa947ec73fe474b71 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.777719 10702 tablet_replica.cc:331] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.778169 10702 raft_consensus.cc:2238] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Raft consensus shutting down.
I20250113 08:57:42.779103 10702 raft_consensus.cc:2267] T 41e954768662450384ee92db7f7a4aa9 P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.780915 10702 tablet_replica.cc:331] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.781423 10702 raft_consensus.cc:2238] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 2 LEADER]: Raft consensus shutting down.
I20250113 08:57:42.782207 10702 raft_consensus.cc:2267] T a20251bc2d314fa4b9e95d1c982716fa P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.783747 10702 tablet_replica.cc:331] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.784247 10702 raft_consensus.cc:2238] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.784718 10702 raft_consensus.cc:2267] T a6a167a8ddd44f4b96728813a14bf98c P b793334f8c5a4a6aa947ec73fe474b71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.786391 10702 tablet_replica.cc:331] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71: stopping tablet replica
I20250113 08:57:42.786815 10702 raft_consensus.cc:2238] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 LEADER]: Raft consensus shutting down.
I20250113 08:57:42.787572 10702 raft_consensus.cc:2267] T 9af2cb63e2c74278a497a485a1631498 P b793334f8c5a4a6aa947ec73fe474b71 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.837381 10702 tablet_server.cc:195] TabletServer@127.10.115.129:0 shutdown complete.
I20250113 08:57:42.860505 10702 tablet_server.cc:178] TabletServer@127.10.115.130:0 shutting down...
W20250113 08:57:42.881908 11473 consensus_peers.cc:487] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 -> Peer b793334f8c5a4a6aa947ec73fe474b71 (127.10.115.129:41471): Couldn't send request to peer b793334f8c5a4a6aa947ec73fe474b71. Status: Network error: Client connection negotiation failed: client connection to 127.10.115.129:41471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:42.890125 10702 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 08:57:42.890813 10702 tablet_replica.cc:331] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.891528 10702 raft_consensus.cc:2238] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.892074 10702 raft_consensus.cc:2267] T a20251bc2d314fa4b9e95d1c982716fa P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.894021 10702 tablet_replica.cc:331] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.894523 10702 raft_consensus.cc:2238] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 4 LEADER]: Raft consensus shutting down.
I20250113 08:57:42.895582 10702 raft_consensus.cc:2267] T feaff5dad2a9474e94aae9e091ffae5a P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.897751 10702 tablet_replica.cc:331] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
W20250113 08:57:42.898350 11473 consensus_peers.cc:487] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 -> Peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Couldn't send request to peer ea0cbc9eb22d4063b00e12349d64f478. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:42.898373 10702 raft_consensus.cc:2238] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.899012 10702 raft_consensus.cc:2267] T d04408dc82344bd2bce7cc2128d21e06 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.901051 10702 tablet_replica.cc:331] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.901657 10702 raft_consensus.cc:2238] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.902129 10702 raft_consensus.cc:2267] T 0f223f52b9b74b86bbde4a36265996ae P ea0cbc9eb22d4063b00e12349d64f478 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.903856 10702 tablet_replica.cc:331] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.904444 10702 raft_consensus.cc:2238] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 5 LEADER]: Raft consensus shutting down.
I20250113 08:57:42.905608 10702 raft_consensus.cc:2267] T 394f7fa139c04737ae3f52f53323923f P ea0cbc9eb22d4063b00e12349d64f478 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.907665 10702 tablet_replica.cc:331] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
W20250113 08:57:42.907723 11473 consensus_peers.cc:487] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 -> Peer ea0cbc9eb22d4063b00e12349d64f478 (127.10.115.130:42101): Couldn't send request to peer ea0cbc9eb22d4063b00e12349d64f478. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:42.908239 10702 raft_consensus.cc:2238] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.908684 10702 raft_consensus.cc:2267] T 41e954768662450384ee92db7f7a4aa9 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.910521 10702 tablet_replica.cc:331] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.911051 10702 raft_consensus.cc:2238] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.911458 10702 raft_consensus.cc:2267] T 4e649934e3bf40ad8bdcf7797d0c13d7 P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.913240 10702 tablet_replica.cc:331] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.913794 10702 raft_consensus.cc:2238] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.914214 10702 raft_consensus.cc:2267] T 05c4365410d14f8d9d3ab81918a2b854 P ea0cbc9eb22d4063b00e12349d64f478 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.915818 10702 tablet_replica.cc:331] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.916288 10702 raft_consensus.cc:2238] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:42.916721 10702 raft_consensus.cc:2267] T 9af2cb63e2c74278a497a485a1631498 P ea0cbc9eb22d4063b00e12349d64f478 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.918351 10702 tablet_replica.cc:331] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478: stopping tablet replica
I20250113 08:57:42.918776 10702 raft_consensus.cc:2238] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 2 LEADER]: Raft consensus shutting down.
I20250113 08:57:42.919492 10702 raft_consensus.cc:2267] T a6a167a8ddd44f4b96728813a14bf98c P ea0cbc9eb22d4063b00e12349d64f478 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:42.963999 10702 tablet_server.cc:195] TabletServer@127.10.115.130:0 shutdown complete.
I20250113 08:57:42.984795 10702 tablet_server.cc:178] TabletServer@127.10.115.131:0 shutting down...
I20250113 08:57:43.010164 10702 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 08:57:43.010866 10702 tablet_replica.cc:331] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.011485 10702 raft_consensus.cc:2238] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:43.011998 10702 raft_consensus.cc:2267] T feaff5dad2a9474e94aae9e091ffae5a P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.013862 10702 tablet_replica.cc:331] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.014433 10702 raft_consensus.cc:2238] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:43.014914 10702 raft_consensus.cc:2267] T 9af2cb63e2c74278a497a485a1631498 P 10711487b8dc43f181e340bd29bc2327 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.016616 10702 tablet_replica.cc:331] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.017102 10702 raft_consensus.cc:2238] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:43.017566 10702 raft_consensus.cc:2267] T a6a167a8ddd44f4b96728813a14bf98c P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.019280 10702 tablet_replica.cc:331] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.019759 10702 raft_consensus.cc:2238] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 2 LEADER]: Raft consensus shutting down.
I20250113 08:57:43.020493 10702 raft_consensus.cc:2267] T 4e649934e3bf40ad8bdcf7797d0c13d7 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.022127 10702 tablet_replica.cc:331] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.022589 10702 raft_consensus.cc:2238] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 4 LEADER]: Raft consensus shutting down.
I20250113 08:57:43.023305 10702 raft_consensus.cc:2267] T 05c4365410d14f8d9d3ab81918a2b854 P 10711487b8dc43f181e340bd29bc2327 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.025009 10702 tablet_replica.cc:331] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.025513 10702 raft_consensus.cc:2238] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:43.025941 10702 raft_consensus.cc:2267] T 41e954768662450384ee92db7f7a4aa9 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.027541 10702 tablet_replica.cc:331] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.028215 10702 raft_consensus.cc:2238] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:43.028616 10702 raft_consensus.cc:2267] T d04408dc82344bd2bce7cc2128d21e06 P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.030256 10702 tablet_replica.cc:331] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.030763 10702 raft_consensus.cc:2238] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:43.031188 10702 raft_consensus.cc:2267] T a20251bc2d314fa4b9e95d1c982716fa P 10711487b8dc43f181e340bd29bc2327 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.032783 10702 tablet_replica.cc:331] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.033188 10702 raft_consensus.cc:2238] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 5 LEADER]: Raft consensus shutting down.
I20250113 08:57:43.033938 10702 raft_consensus.cc:2267] T 0f223f52b9b74b86bbde4a36265996ae P 10711487b8dc43f181e340bd29bc2327 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.035694 10702 tablet_replica.cc:331] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327: stopping tablet replica
I20250113 08:57:43.036087 10702 raft_consensus.cc:2238] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250113 08:57:43.036466 10702 raft_consensus.cc:2267] T 394f7fa139c04737ae3f52f53323923f P 10711487b8dc43f181e340bd29bc2327 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.080627 10702 tablet_server.cc:195] TabletServer@127.10.115.131:0 shutdown complete.
I20250113 08:57:43.101379 10702 master.cc:537] Master@127.10.115.190:41507 shutting down...
I20250113 08:57:43.123226 10702 raft_consensus.cc:2238] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 1 LEADER]: Raft consensus shutting down.
I20250113 08:57:43.123819 10702 raft_consensus.cc:2267] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:57:43.124089 10702 tablet_replica.cc:331] T 00000000000000000000000000000000 P daca8b8de715409bb686ad7e2f5c98ae: stopping tablet replica
I20250113 08:57:43.144153 10702 master.cc:559] Master@127.10.115.190:41507 shutdown complete.
10711487b8dc43f181e340bd29bc2327 1
b793334f8c5a4a6aa947ec73fe474b71 5
ea0cbc9eb22d4063b00e12349d64f478 4
10711487b8dc43f181e340bd29bc2327 3
b793334f8c5a4a6aa947ec73fe474b71 3
ea0cbc9eb22d4063b00e12349d64f478 4
10711487b8dc43f181e340bd29bc2327 1
b793334f8c5a4a6aa947ec73fe474b71 8
ea0cbc9eb22d4063b00e12349d64f478 1
10711487b8dc43f181e340bd29bc2327 3
b793334f8c5a4a6aa947ec73fe474b71 4
ea0cbc9eb22d4063b00e12349d64f478 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29243 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250113 08:57:43.189006 10702 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.115.190:41823
I20250113 08:57:43.190224 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:43.196065 11766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:43.196209 11767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:43.198009 11769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:43.198175 10702 server_base.cc:1034] running on GCE node
I20250113 08:57:43.199234 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:43.199431 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:43.199569 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758663199552 us; error 0 us; skew 500 ppm
I20250113 08:57:43.200065 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:43.208343 10702 webserver.cc:458] Webserver started at http://127.10.115.190:36931/ using document root <none> and password file <none>
I20250113 08:57:43.208882 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:43.209069 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:43.209367 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:43.210613 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/master-0-root/instance:
uuid: "48f8007a91af4c0c837ee737c6326d99"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.215344 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250113 08:57:43.218837 11774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.219592 10702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250113 08:57:43.219893 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/master-0-root
uuid: "48f8007a91af4c0c837ee737c6326d99"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.220166 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:43.233922 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:43.235100 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:43.272159 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.190:41823
I20250113 08:57:43.272244 11825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.190:41823 every 8 connection(s)
I20250113 08:57:43.275928 11826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.285634 11826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Bootstrap starting.
I20250113 08:57:43.290141 11826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:43.294404 11826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: No bootstrap required, opened a new log
I20250113 08:57:43.296510 11826 raft_consensus.cc:357] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f8007a91af4c0c837ee737c6326d99" member_type: VOTER }
I20250113 08:57:43.296985 11826 raft_consensus.cc:383] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:43.297163 11826 raft_consensus.cc:738] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f8007a91af4c0c837ee737c6326d99, State: Initialized, Role: FOLLOWER
I20250113 08:57:43.297756 11826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f8007a91af4c0c837ee737c6326d99" member_type: VOTER }
I20250113 08:57:43.298182 11826 raft_consensus.cc:397] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 08:57:43.298379 11826 raft_consensus.cc:491] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 08:57:43.298638 11826 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:43.302735 11826 raft_consensus.cc:513] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f8007a91af4c0c837ee737c6326d99" member_type: VOTER }
I20250113 08:57:43.303193 11826 leader_election.cc:304] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f8007a91af4c0c837ee737c6326d99; no voters:
I20250113 08:57:43.304311 11826 leader_election.cc:290] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 08:57:43.304702 11829 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:43.306058 11829 raft_consensus.cc:695] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [term 1 LEADER]: Becoming Leader. State: Replica: 48f8007a91af4c0c837ee737c6326d99, State: Running, Role: LEADER
I20250113 08:57:43.306670 11829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f8007a91af4c0c837ee737c6326d99" member_type: VOTER }
I20250113 08:57:43.307435 11826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 08:57:43.312423 11831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f8007a91af4c0c837ee737c6326d99. Latest consensus state: current_term: 1 leader_uuid: "48f8007a91af4c0c837ee737c6326d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f8007a91af4c0c837ee737c6326d99" member_type: VOTER } }
I20250113 08:57:43.313170 11831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [sys.catalog]: This master's current role is: LEADER
I20250113 08:57:43.314093 11830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f8007a91af4c0c837ee737c6326d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f8007a91af4c0c837ee737c6326d99" member_type: VOTER } }
I20250113 08:57:43.314710 11830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99 [sys.catalog]: This master's current role is: LEADER
I20250113 08:57:43.315181 11838 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 08:57:43.321486 11838 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 08:57:43.322664 10702 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 08:57:43.330291 11838 catalog_manager.cc:1348] Generated new cluster ID: f4f30d5c9b4943ec8014949ebd9a9b6d
I20250113 08:57:43.330605 11838 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 08:57:43.352277 11838 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 08:57:43.353585 11838 catalog_manager.cc:1505] Loading token signing keys...
I20250113 08:57:43.366720 11838 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Generated new TSK 0
I20250113 08:57:43.367340 11838 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 08:57:43.389590 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:43.395690 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:43.396725 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:43.398308 10702 server_base.cc:1034] running on GCE node
W20250113 08:57:43.398836 11850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:43.399695 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:43.399901 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:43.400060 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758663400041 us; error 0 us; skew 500 ppm
I20250113 08:57:43.400573 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:43.403075 10702 webserver.cc:458] Webserver started at http://127.10.115.129:40409/ using document root <none> and password file <none>
I20250113 08:57:43.403540 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:43.403724 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:43.403990 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:43.405162 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/instance:
uuid: "6ea00811b7d44b49b872ac5c03b91487"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.409847 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250113 08:57:43.413218 11855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.414147 10702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250113 08:57:43.414616 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root
uuid: "6ea00811b7d44b49b872ac5c03b91487"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.415187 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:43.431638 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:43.432946 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:43.434515 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:57:43.436897 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:57:43.437084 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.437306 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:57:43.437453 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.479755 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.129:33539
I20250113 08:57:43.479872 11917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.129:33539 every 8 connection(s)
I20250113 08:57:43.484843 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:43.491681 11922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:43.493291 11923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:43.495620 10702 server_base.cc:1034] running on GCE node
W20250113 08:57:43.496120 11925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:43.497326 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:43.497669 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:43.497898 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758663497878 us; error 0 us; skew 500 ppm
I20250113 08:57:43.498492 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:43.498771 11918 heartbeater.cc:346] Connected to a master server at 127.10.115.190:41823
I20250113 08:57:43.499202 11918 heartbeater.cc:463] Registering TS with master...
I20250113 08:57:43.500159 11918 heartbeater.cc:510] Master 127.10.115.190:41823 requested a full tablet report, sending...
I20250113 08:57:43.501194 10702 webserver.cc:458] Webserver started at http://127.10.115.130:44943/ using document root <none> and password file <none>
I20250113 08:57:43.501753 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:43.501937 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:43.502172 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:43.502434 11791 ts_manager.cc:194] Registered new tserver with Master: 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:57:43.503365 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/instance:
uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.504280 11791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44354
I20250113 08:57:43.507961 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250113 08:57:43.511413 11930 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.512164 10702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250113 08:57:43.512427 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root
uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.512714 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:43.521646 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:43.522886 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:43.524322 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:57:43.526538 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:57:43.526741 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.526969 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:57:43.527120 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.566326 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.130:41863
I20250113 08:57:43.566459 11992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.130:41863 every 8 connection(s)
I20250113 08:57:43.571105 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:57:43.578439 11996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:43.580598 11997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:57:43.584357 11999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:57:43.585233 10702 server_base.cc:1034] running on GCE node
I20250113 08:57:43.585786 11993 heartbeater.cc:346] Connected to a master server at 127.10.115.190:41823
I20250113 08:57:43.586252 11993 heartbeater.cc:463] Registering TS with master...
I20250113 08:57:43.586614 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:57:43.586834 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:57:43.586977 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758663586966 us; error 0 us; skew 500 ppm
I20250113 08:57:43.587051 11993 heartbeater.cc:510] Master 127.10.115.190:41823 requested a full tablet report, sending...
I20250113 08:57:43.587524 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:57:43.591773 11791 ts_manager.cc:194] Registered new tserver with Master: 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:43.593108 10702 webserver.cc:458] Webserver started at http://127.10.115.131:42565/ using document root <none> and password file <none>
I20250113 08:57:43.593802 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:57:43.593878 11791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44358
I20250113 08:57:43.594117 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:57:43.594456 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:57:43.597005 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/instance:
uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.610399 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.013s user 0.010s sys 0.000s
I20250113 08:57:43.616518 12004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.618747 10702 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250113 08:57:43.619374 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root
uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
format_stamp: "Formatted at 2025-01-13 08:57:43 on dist-test-slave-lrhs"
I20250113 08:57:43.620047 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:57:43.656206 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:57:43.657567 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:57:43.659020 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:57:43.661238 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:57:43.661458 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.661695 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:57:43.661847 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:57:43.701709 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.131:43231
I20250113 08:57:43.701807 12066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.131:43231 every 8 connection(s)
I20250113 08:57:43.714752 12067 heartbeater.cc:346] Connected to a master server at 127.10.115.190:41823
I20250113 08:57:43.715126 12067 heartbeater.cc:463] Registering TS with master...
I20250113 08:57:43.715837 12067 heartbeater.cc:510] Master 127.10.115.190:41823 requested a full tablet report, sending...
I20250113 08:57:43.717758 11791 ts_manager.cc:194] Registered new tserver with Master: 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:43.718390 10702 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013651556s
I20250113 08:57:43.719525 11791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60016
I20250113 08:57:43.721220 10702 test_util.cc:274] Using random seed: -1556729621
I20250113 08:57:43.745735 11791 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60030:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 08:57:43.751987 11791 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 08:57:43.955776 11883 tablet_service.cc:1467] Processing CreateTablet for tablet b6a40dc5e61345558c943df4be928468 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250113 08:57:43.957281 11883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6a40dc5e61345558c943df4be928468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.957824 11882 tablet_service.cc:1467] Processing CreateTablet for tablet 69aa379773c047589ae2362850997a47 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250113 08:57:43.959954 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 79302a9220a149cb81cb2573664290a1 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250113 08:57:43.962026 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 099cf849e7414101a8741339c72fa73e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250113 08:57:43.962455 11958 tablet_service.cc:1467] Processing CreateTablet for tablet b6a40dc5e61345558c943df4be928468 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250113 08:57:43.963776 11958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6a40dc5e61345558c943df4be928468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.964072 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 54d2a9b9e9134754acb8568b23bba884 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250113 08:57:43.966218 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 272c58aa3da64d86aa8a3e3890e3a249 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250113 08:57:43.966415 11957 tablet_service.cc:1467] Processing CreateTablet for tablet 69aa379773c047589ae2362850997a47 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250113 08:57:43.967705 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69aa379773c047589ae2362850997a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.968284 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 39676b99e4434f11b6017bb867fb7471 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250113 08:57:43.970024 11882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69aa379773c047589ae2362850997a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.970466 11956 tablet_service.cc:1467] Processing CreateTablet for tablet 39676b99e4434f11b6017bb867fb7471 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250113 08:57:43.971827 11956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39676b99e4434f11b6017bb867fb7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.975092 11955 tablet_service.cc:1467] Processing CreateTablet for tablet 79302a9220a149cb81cb2573664290a1 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250113 08:57:43.976774 11955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79302a9220a149cb81cb2573664290a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.978718 11954 tablet_service.cc:1467] Processing CreateTablet for tablet 099cf849e7414101a8741339c72fa73e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250113 08:57:43.980124 11954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099cf849e7414101a8741339c72fa73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.982445 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 54d2a9b9e9134754acb8568b23bba884 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250113 08:57:43.983114 11950 tablet_service.cc:1467] Processing CreateTablet for tablet 18dede64eda24dff8bfe2f979a4115d5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250113 08:57:43.983906 11953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d2a9b9e9134754acb8568b23bba884. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.985073 11950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dede64eda24dff8bfe2f979a4115d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.986464 11951 tablet_service.cc:1467] Processing CreateTablet for tablet 67a8b82ef293439e815f49903f8e5dbb (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250113 08:57:43.987848 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67a8b82ef293439e815f49903f8e5dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.989982 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 18dede64eda24dff8bfe2f979a4115d5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250113 08:57:43.990417 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 7dda9aa8769e4146a4dabeadd588c734 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250113 08:57:43.991430 12024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dede64eda24dff8bfe2f979a4115d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.991709 11949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dda9aa8769e4146a4dabeadd588c734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.994755 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 39676b99e4434f11b6017bb867fb7471 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250113 08:57:44.017663 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39676b99e4434f11b6017bb867fb7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.999382 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 67a8b82ef293439e815f49903f8e5dbb (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250113 08:57:44.019774 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67a8b82ef293439e815f49903f8e5dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.004648 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 69aa379773c047589ae2362850997a47 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250113 08:57:44.005091 11881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39676b99e4434f11b6017bb867fb7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.009929 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 272c58aa3da64d86aa8a3e3890e3a249 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250113 08:57:44.021492 12031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69aa379773c047589ae2362850997a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.022226 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272c58aa3da64d86aa8a3e3890e3a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.012672 12032 tablet_service.cc:1467] Processing CreateTablet for tablet b6a40dc5e61345558c943df4be928468 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250113 08:57:44.023365 12032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6a40dc5e61345558c943df4be928468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.013502 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 7dda9aa8769e4146a4dabeadd588c734 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250113 08:57:44.025281 12023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dda9aa8769e4146a4dabeadd588c734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.007624 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 099cf849e7414101a8741339c72fa73e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250113 08:57:44.027081 12028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099cf849e7414101a8741339c72fa73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.008822 12029 tablet_service.cc:1467] Processing CreateTablet for tablet 79302a9220a149cb81cb2573664290a1 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250113 08:57:44.028877 12029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79302a9220a149cb81cb2573664290a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.006433 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 54d2a9b9e9134754acb8568b23bba884 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250113 08:57:44.031001 12027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d2a9b9e9134754acb8568b23bba884. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:43.997162 12026 tablet_service.cc:1467] Processing CreateTablet for tablet 272c58aa3da64d86aa8a3e3890e3a249 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250113 08:57:44.024225 11880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79302a9220a149cb81cb2573664290a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.032814 12026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272c58aa3da64d86aa8a3e3890e3a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.033047 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272c58aa3da64d86aa8a3e3890e3a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.034085 11878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d2a9b9e9134754acb8568b23bba884. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.035297 11879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099cf849e7414101a8741339c72fa73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.099334 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 67a8b82ef293439e815f49903f8e5dbb (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250113 08:57:44.101310 11876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67a8b82ef293439e815f49903f8e5dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.110569 11874 tablet_service.cc:1467] Processing CreateTablet for tablet 7dda9aa8769e4146a4dabeadd588c734 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250113 08:57:44.112741 11874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dda9aa8769e4146a4dabeadd588c734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.113394 11875 tablet_service.cc:1467] Processing CreateTablet for tablet 18dede64eda24dff8bfe2f979a4115d5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250113 08:57:44.114848 11875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dede64eda24dff8bfe2f979a4115d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.210537 11953 tablet_service.cc:1467] Processing CreateTablet for tablet cee55da481c64a6d885aa3ab442761cd (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250113 08:57:44.226096 12027 tablet_service.cc:1467] Processing CreateTablet for tablet cee55da481c64a6d885aa3ab442761cd (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250113 08:57:44.227991 12027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee55da481c64a6d885aa3ab442761cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.229712 11880 tablet_service.cc:1467] Processing CreateTablet for tablet cee55da481c64a6d885aa3ab442761cd (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250113 08:57:44.231127 11880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee55da481c64a6d885aa3ab442761cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.257884 12082 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:44.255963 11953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee55da481c64a6d885aa3ab442761cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.261788 12083 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:44.261680 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe494493d33439a9e129bc5d8ab6654 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250113 08:57:44.263111 12028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe494493d33439a9e129bc5d8ab6654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.262984 11958 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6bba22615448a8acbbb898fb381116 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250113 08:57:44.265681 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe494493d33439a9e129bc5d8ab6654 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250113 08:57:44.267472 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6bba22615448a8acbbb898fb381116 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250113 08:57:44.268684 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe494493d33439a9e129bc5d8ab6654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.270224 11951 tablet_service.cc:1467] Processing CreateTablet for tablet badac2289c8b4a10915cca20182fc423 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250113 08:57:44.268692 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6bba22615448a8acbbb898fb381116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.271941 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet badac2289c8b4a10915cca20182fc423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.276057 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6bba22615448a8acbbb898fb381116 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250113 08:57:44.303434 11876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6bba22615448a8acbbb898fb381116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.304211 11958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6bba22615448a8acbbb898fb381116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.276621 11875 tablet_service.cc:1467] Processing CreateTablet for tablet 8f63a7b00ea546f2865a4b5d730e35de (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250113 08:57:44.306105 11875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f63a7b00ea546f2865a4b5d730e35de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.277941 12084 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:44.284276 12026 tablet_service.cc:1467] Processing CreateTablet for tablet 8f63a7b00ea546f2865a4b5d730e35de (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250113 08:57:44.284874 12031 tablet_service.cc:1467] Processing CreateTablet for tablet badac2289c8b4a10915cca20182fc423 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250113 08:57:44.291975 12082 tablet_bootstrap.cc:654] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:44.293107 12083 tablet_bootstrap.cc:654] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:44.268036 11954 tablet_service.cc:1467] Processing CreateTablet for tablet 8f63a7b00ea546f2865a4b5d730e35de (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250113 08:57:44.295862 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1a8409abe745918b8f11dc8590376a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250113 08:57:44.314328 12031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet badac2289c8b4a10915cca20182fc423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.315148 12032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1a8409abe745918b8f11dc8590376a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.314363 11954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f63a7b00ea546f2865a4b5d730e35de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.321282 12026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f63a7b00ea546f2865a4b5d730e35de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.288704 12024 tablet_service.cc:1467] Processing CreateTablet for tablet e2677547aaee4bba86cf339fe13278bf (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250113 08:57:44.297650 11883 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe494493d33439a9e129bc5d8ab6654 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250113 08:57:44.298296 11877 tablet_service.cc:1467] Processing CreateTablet for tablet badac2289c8b4a10915cca20182fc423 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250113 08:57:44.300778 11957 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1a8409abe745918b8f11dc8590376a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250113 08:57:44.324524 12024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2677547aaee4bba86cf339fe13278bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.323529 11883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe494493d33439a9e129bc5d8ab6654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.327908 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet badac2289c8b4a10915cca20182fc423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.324681 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1a8409abe745918b8f11dc8590376a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 08:57:44.340940 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:44.351069 11956 tablet_service.cc:1467] Processing CreateTablet for tablet a23cb6607d3a463293b397049ef2d690 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250113 08:57:44.352413 11956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a23cb6607d3a463293b397049ef2d690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.353546 11950 tablet_service.cc:1467] Processing CreateTablet for tablet e2677547aaee4bba86cf339fe13278bf (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250113 08:57:44.354766 11950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2677547aaee4bba86cf339fe13278bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.363842 12082 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:44.364356 12082 ts_tablet_manager.cc:1397] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.004s
I20250113 08:57:44.365746 11949 tablet_service.cc:1467] Processing CreateTablet for tablet fcc31099ea8040b7bfbfe7d779148ca5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250113 08:57:44.367197 11949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc31099ea8040b7bfbfe7d779148ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.369544 12083 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:44.370047 12083 ts_tablet_manager.cc:1397] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.109s user 0.009s sys 0.013s
I20250113 08:57:44.374943 11881 tablet_service.cc:1467] Processing CreateTablet for tablet e2677547aaee4bba86cf339fe13278bf (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250113 08:57:44.376236 11881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2677547aaee4bba86cf339fe13278bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.376806 11882 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1a8409abe745918b8f11dc8590376a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250113 08:57:44.378026 11882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1a8409abe745918b8f11dc8590376a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.380159 11955 tablet_service.cc:1467] Processing CreateTablet for tablet a43de3c8f06b48358bcee58f9567bb3b (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250113 08:57:44.388374 11874 tablet_service.cc:1467] Processing CreateTablet for tablet fcc31099ea8040b7bfbfe7d779148ca5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250113 08:57:44.372818 12083 raft_consensus.cc:357] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.389703 12083 raft_consensus.cc:383] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:44.390053 12083 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:44.380844 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 80cc6c97986444348b1837c27c41338a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250113 08:57:44.391942 12083 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.367254 12082 raft_consensus.cc:357] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.393919 12082 raft_consensus.cc:383] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:44.394215 12082 raft_consensus.cc:738] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:44.394984 12082 consensus_queue.cc:260] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.396735 11874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc31099ea8040b7bfbfe7d779148ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.397702 12023 tablet_service.cc:1467] Processing CreateTablet for tablet fcc31099ea8040b7bfbfe7d779148ca5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250113 08:57:44.399075 12023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc31099ea8040b7bfbfe7d779148ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.388366 11955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43de3c8f06b48358bcee58f9567bb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.360891 12029 tablet_service.cc:1467] Processing CreateTablet for tablet a23cb6607d3a463293b397049ef2d690 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250113 08:57:44.409601 11953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc6c97986444348b1837c27c41338a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.386994 11880 tablet_service.cc:1467] Processing CreateTablet for tablet a23cb6607d3a463293b397049ef2d690 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250113 08:57:44.409590 12029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a23cb6607d3a463293b397049ef2d690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.345142 12084 tablet_bootstrap.cc:654] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:44.439714 11880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a23cb6607d3a463293b397049ef2d690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.445940 11874 tablet_service.cc:1467] Processing CreateTablet for tablet 80cc6c97986444348b1837c27c41338a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250113 08:57:44.447319 11874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc6c97986444348b1837c27c41338a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.461017 11877 tablet_service.cc:1467] Processing CreateTablet for tablet a43de3c8f06b48358bcee58f9567bb3b (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250113 08:57:44.462440 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43de3c8f06b48358bcee58f9567bb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.453670 11879 tablet_service.cc:1467] Processing CreateTablet for tablet fc84bb347ec2413e9188b7253b0812ab (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250113 08:57:44.418877 12030 tablet_service.cc:1467] Processing CreateTablet for tablet a43de3c8f06b48358bcee58f9567bb3b (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250113 08:57:44.476874 12067 heartbeater.cc:502] Master 127.10.115.190:41823 was elected leader, sending a full tablet report...
I20250113 08:57:44.479167 12082 ts_tablet_manager.cc:1428] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.114s user 0.019s sys 0.009s
I20250113 08:57:44.482132 11918 heartbeater.cc:502] Master 127.10.115.190:41823 was elected leader, sending a full tablet report...
I20250113 08:57:44.493125 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43de3c8f06b48358bcee58f9567bb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.509966 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 80cc6c97986444348b1837c27c41338a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250113 08:57:44.511344 12027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc6c97986444348b1837c27c41338a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.512195 11951 tablet_service.cc:1467] Processing CreateTablet for tablet 705a99b7862446be849d4c843d8084ee (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250113 08:57:44.513581 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705a99b7862446be849d4c843d8084ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.514194 12083 ts_tablet_manager.cc:1428] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.144s user 0.018s sys 0.011s
I20250113 08:57:44.515256 12083 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:44.514226 11955 tablet_service.cc:1467] Processing CreateTablet for tablet 992519c5663649019a1ff9753075ddc7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250113 08:57:44.518213 11955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992519c5663649019a1ff9753075ddc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.522145 11952 tablet_service.cc:1467] Processing CreateTablet for tablet ce0640fd1a5740f1ac76ebcae33dfa25 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250113 08:57:44.523603 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0640fd1a5740f1ac76ebcae33dfa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.532296 11958 tablet_service.cc:1467] Processing CreateTablet for tablet a2bbae0b61a948339e4c443b9237ad87 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250113 08:57:44.532594 11951 tablet_service.cc:1467] Processing CreateTablet for tablet 54973520c9884989a12742d4eb45994f (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250113 08:57:44.533691 11958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bbae0b61a948339e4c443b9237ad87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.533756 11883 tablet_service.cc:1467] Processing CreateTablet for tablet ce0640fd1a5740f1ac76ebcae33dfa25 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250113 08:57:44.534476 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54973520c9884989a12742d4eb45994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.534914 11957 tablet_service.cc:1467] Processing CreateTablet for tablet fc84bb347ec2413e9188b7253b0812ab (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250113 08:57:44.536116 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc84bb347ec2413e9188b7253b0812ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.543051 12025 tablet_service.cc:1467] Processing CreateTablet for tablet fc84bb347ec2413e9188b7253b0812ab (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250113 08:57:44.544391 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc84bb347ec2413e9188b7253b0812ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.560254 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 992519c5663649019a1ff9753075ddc7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250113 08:57:44.562387 11878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992519c5663649019a1ff9753075ddc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.562453 11951 tablet_service.cc:1467] Processing CreateTablet for tablet 588088c74b20488da56f01a0b351a918 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250113 08:57:44.563884 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588088c74b20488da56f01a0b351a918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.571110 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 992519c5663649019a1ff9753075ddc7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250113 08:57:44.572103 12087 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:44.572494 12032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992519c5663649019a1ff9753075ddc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.552659 12031 tablet_service.cc:1467] Processing CreateTablet for tablet ce0640fd1a5740f1ac76ebcae33dfa25 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250113 08:57:44.575222 11883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0640fd1a5740f1ac76ebcae33dfa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.572583 12087 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.576958 12084 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:44.577430 12084 ts_tablet_manager.cc:1397] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.300s user 0.026s sys 0.037s
I20250113 08:57:44.577961 12087 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:44.578130 12082 tablet_bootstrap.cc:492] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:44.578701 11875 tablet_service.cc:1467] Processing CreateTablet for tablet a2bbae0b61a948339e4c443b9237ad87 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250113 08:57:44.579970 11875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bbae0b61a948339e4c443b9237ad87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.580106 12084 raft_consensus.cc:357] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.580868 12084 raft_consensus.cc:383] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:44.584371 12082 tablet_bootstrap.cc:654] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:44.591707 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 705a99b7862446be849d4c843d8084ee (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250113 08:57:44.592152 12023 tablet_service.cc:1467] Processing CreateTablet for tablet a2bbae0b61a948339e4c443b9237ad87 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250113 08:57:44.593080 12028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705a99b7862446be849d4c843d8084ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.599123 11993 heartbeater.cc:502] Master 127.10.115.190:41823 was elected leader, sending a full tablet report...
I20250113 08:57:44.593017 11882 tablet_service.cc:1467] Processing CreateTablet for tablet 54973520c9884989a12742d4eb45994f (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250113 08:57:44.605419 11882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54973520c9884989a12742d4eb45994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.581138 12084 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:44.606439 12084 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.610846 11954 tablet_service.cc:1467] Processing CreateTablet for tablet faa96abb4c9940e4aa27327deeb68af0 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250113 08:57:44.612244 11954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa96abb4c9940e4aa27327deeb68af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.610873 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 49d70ffc378340e785c4f8dde909f9ba (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250113 08:57:44.614629 11881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d70ffc378340e785c4f8dde909f9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.599820 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 49d70ffc378340e785c4f8dde909f9ba (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250113 08:57:44.618507 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d70ffc378340e785c4f8dde909f9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.624641 12023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bbae0b61a948339e4c443b9237ad87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.635136 11874 tablet_service.cc:1467] Processing CreateTablet for tablet 705a99b7862446be849d4c843d8084ee (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250113 08:57:44.636617 11957 tablet_service.cc:1467] Processing CreateTablet for tablet a47a39083de44711b040bcf422953893 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250113 08:57:44.637184 11879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc84bb347ec2413e9188b7253b0812ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.653229 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 588088c74b20488da56f01a0b351a918 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250113 08:57:44.644318 12026 tablet_service.cc:1467] Processing CreateTablet for tablet 588088c74b20488da56f01a0b351a918 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250113 08:57:44.654470 12031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0640fd1a5740f1ac76ebcae33dfa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.642851 12029 tablet_service.cc:1467] Processing CreateTablet for tablet 54973520c9884989a12742d4eb45994f (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250113 08:57:44.655836 12029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54973520c9884989a12742d4eb45994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.657737 11874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705a99b7862446be849d4c843d8084ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.657666 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 49d70ffc378340e785c4f8dde909f9ba (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250113 08:57:44.658960 12024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d70ffc378340e785c4f8dde909f9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.661890 12026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588088c74b20488da56f01a0b351a918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.664462 11876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588088c74b20488da56f01a0b351a918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.664187 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 364888f946914390a5a07be6735a4cc9 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250113 08:57:44.666880 11950 tablet_service.cc:1467] Processing CreateTablet for tablet b300f59ff3964ee28bb6ef0c1141395e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250113 08:57:44.665310 12083 tablet_bootstrap.cc:654] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:44.672667 12084 ts_tablet_manager.cc:1428] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.095s user 0.014s sys 0.005s
I20250113 08:57:44.673362 12027 tablet_service.cc:1467] Processing CreateTablet for tablet faa96abb4c9940e4aa27327deeb68af0 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250113 08:57:44.674991 12027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa96abb4c9940e4aa27327deeb68af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.694366 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47a39083de44711b040bcf422953893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.702209 11880 tablet_service.cc:1467] Processing CreateTablet for tablet faa96abb4c9940e4aa27327deeb68af0 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250113 08:57:44.703799 11880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa96abb4c9940e4aa27327deeb68af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.701023 11877 tablet_service.cc:1467] Processing CreateTablet for tablet a47a39083de44711b040bcf422953893 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250113 08:57:44.714787 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47a39083de44711b040bcf422953893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.716682 12030 tablet_service.cc:1467] Processing CreateTablet for tablet a47a39083de44711b040bcf422953893 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250113 08:57:44.718739 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47a39083de44711b040bcf422953893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.738401 12025 tablet_service.cc:1467] Processing CreateTablet for tablet b300f59ff3964ee28bb6ef0c1141395e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250113 08:57:44.739892 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b300f59ff3964ee28bb6ef0c1141395e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.743064 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:44.746860 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:44.751406 12084 tablet_bootstrap.cc:492] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
W20250113 08:57:44.751786 12007 leader_election.cc:343] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:44.752841 12006 leader_election.cc:343] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:44.753233 12006 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:44.754716 12087 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:44.756685 11950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b300f59ff3964ee28bb6ef0c1141395e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.764573 11955 tablet_service.cc:1467] Processing CreateTablet for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250113 08:57:44.769762 11949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364888f946914390a5a07be6735a4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.775120 11951 tablet_service.cc:1467] Processing CreateTablet for tablet 9a457bc5206c46d7b198dbff2b1c137c (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250113 08:57:44.776472 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a457bc5206c46d7b198dbff2b1c137c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.778681 11883 tablet_service.cc:1467] Processing CreateTablet for tablet b300f59ff3964ee28bb6ef0c1141395e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250113 08:57:44.779661 11955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.779981 11883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b300f59ff3964ee28bb6ef0c1141395e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.782243 11956 tablet_service.cc:1467] Processing CreateTablet for tablet 663c56b937924e7abebd7025ca892c1e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250113 08:57:44.795621 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 1527f41e7e954abc8622a5f5da292602 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250113 08:57:44.796554 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 364888f946914390a5a07be6735a4cc9 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250113 08:57:44.800293 12084 tablet_bootstrap.cc:654] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:44.801815 11878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364888f946914390a5a07be6735a4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.809230 11952 tablet_service.cc:1467] Processing CreateTablet for tablet e9223bd4b75148bc8c6a0349dcc26b9b (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250113 08:57:44.810124 11953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1527f41e7e954abc8622a5f5da292602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.810725 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9223bd4b75148bc8c6a0349dcc26b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.812748 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250113 08:57:44.814476 12031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.817351 11956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663c56b937924e7abebd7025ca892c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.820741 12083 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:44.821514 12083 ts_tablet_manager.cc:1397] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.306s user 0.007s sys 0.019s
I20250113 08:57:44.820573 11882 tablet_service.cc:1467] Processing CreateTablet for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250113 08:57:44.822608 11882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.824772 12083 raft_consensus.cc:357] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.828547 11958 tablet_service.cc:1467] Processing CreateTablet for tablet de29a6701ddb4906a7eff4d34bf97ab5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250113 08:57:44.832335 11958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de29a6701ddb4906a7eff4d34bf97ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.832568 12082 tablet_bootstrap.cc:492] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:44.853698 12082 ts_tablet_manager.cc:1397] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.276s user 0.009s sys 0.018s
I20250113 08:57:44.856616 12082 raft_consensus.cc:357] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.857632 12082 raft_consensus.cc:383] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:44.858003 12082 raft_consensus.cc:738] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:44.803565 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 364888f946914390a5a07be6735a4cc9 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250113 08:57:44.835839 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 9a457bc5206c46d7b198dbff2b1c137c (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250113 08:57:44.859781 11879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a457bc5206c46d7b198dbff2b1c137c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.836812 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 663c56b937924e7abebd7025ca892c1e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250113 08:57:44.841629 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 9a457bc5206c46d7b198dbff2b1c137c (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250113 08:57:44.864410 12082 consensus_queue.cc:260] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.846912 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 663c56b937924e7abebd7025ca892c1e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250113 08:57:44.850354 12029 tablet_service.cc:1467] Processing CreateTablet for tablet 1527f41e7e954abc8622a5f5da292602 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250113 08:57:44.850425 12032 tablet_service.cc:1467] Processing CreateTablet for tablet e9223bd4b75148bc8c6a0349dcc26b9b (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250113 08:57:44.851109 11954 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1ace8d6d3d488583610962dabdee94 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250113 08:57:44.862043 12024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364888f946914390a5a07be6735a4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.886039 11954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1ace8d6d3d488583610962dabdee94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.862010 11874 tablet_service.cc:1467] Processing CreateTablet for tablet 1527f41e7e954abc8622a5f5da292602 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250113 08:57:44.851112 12083 raft_consensus.cc:383] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:44.861955 12023 tablet_service.cc:1467] Processing CreateTablet for tablet de29a6701ddb4906a7eff4d34bf97ab5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250113 08:57:44.890200 12023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de29a6701ddb4906a7eff4d34bf97ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.893985 12088 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:44.894454 12088 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.897397 12083 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:44.898551 12083 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.899273 12088 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:44.877871 11881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663c56b937924e7abebd7025ca892c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.876078 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 531b33203b294592b3a1275dc74ab4f3 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250113 08:57:44.909169 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a457bc5206c46d7b198dbff2b1c137c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.909947 12082 ts_tablet_manager.cc:1428] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.056s user 0.006s sys 0.002s
I20250113 08:57:44.913528 12082 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:44.917850 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 531b33203b294592b3a1275dc74ab4f3 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250113 08:57:44.919175 12031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531b33203b294592b3a1275dc74ab4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.919956 12083 ts_tablet_manager.cc:1428] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.098s user 0.006s sys 0.002s
I20250113 08:57:44.920926 12083 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:44.926414 11876 tablet_service.cc:1467] Processing CreateTablet for tablet e9223bd4b75148bc8c6a0349dcc26b9b (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250113 08:57:44.927369 11878 tablet_service.cc:1467] Processing CreateTablet for tablet de29a6701ddb4906a7eff4d34bf97ab5 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250113 08:57:44.927824 11876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9223bd4b75148bc8c6a0349dcc26b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.928606 11878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de29a6701ddb4906a7eff4d34bf97ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.934391 11883 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1ace8d6d3d488583610962dabdee94 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250113 08:57:44.934612 11950 tablet_service.cc:1467] Processing CreateTablet for tablet 7104dce7b01a423493a37b6ed694da07 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250113 08:57:44.935798 11883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1ace8d6d3d488583610962dabdee94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.936434 11949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531b33203b294592b3a1275dc74ab4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.937455 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1ace8d6d3d488583610962dabdee94 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250113 08:57:44.938670 12028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1ace8d6d3d488583610962dabdee94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.945335 12032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9223bd4b75148bc8c6a0349dcc26b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.947419 12027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663c56b937924e7abebd7025ca892c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.948490 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 531b33203b294592b3a1275dc74ab4f3 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250113 08:57:44.949342 12029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1527f41e7e954abc8622a5f5da292602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.950287 11880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531b33203b294592b3a1275dc74ab4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.974243 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 8210eb7687124bcfbc64e314e532799e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250113 08:57:44.975601 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8210eb7687124bcfbc64e314e532799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.977514 11874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1527f41e7e954abc8622a5f5da292602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.982002 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7522e543294e28ac66b2faf6001cbe (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250113 08:57:44.983273 12024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7522e543294e28ac66b2faf6001cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:44.986716 11875 tablet_service.cc:1467] Processing CreateTablet for tablet 7104dce7b01a423493a37b6ed694da07 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250113 08:57:44.989535 12093 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:44.990006 12093 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:44.993981 12026 tablet_service.cc:1467] Processing CreateTablet for tablet 7104dce7b01a423493a37b6ed694da07 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250113 08:57:45.005118 12084 tablet_bootstrap.cc:492] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.005700 12084 ts_tablet_manager.cc:1397] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.255s user 0.015s sys 0.005s
I20250113 08:57:45.008539 12084 raft_consensus.cc:357] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.009653 11957 tablet_service.cc:1467] Processing CreateTablet for tablet 8210eb7687124bcfbc64e314e532799e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250113 08:57:45.012779 12093 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.018474 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 8210eb7687124bcfbc64e314e532799e (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250113 08:57:45.019807 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8210eb7687124bcfbc64e314e532799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.010747 12084 raft_consensus.cc:383] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.021502 12084 raft_consensus.cc:738] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.025835 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8210eb7687124bcfbc64e314e532799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.040181 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:45.044780 11858 leader_election.cc:343] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.046991 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7522e543294e28ac66b2faf6001cbe (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250113 08:57:45.047402 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 218ea9985e1d40a48892a8e01aaaef39 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250113 08:57:45.047787 11951 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7522e543294e28ac66b2faf6001cbe (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250113 08:57:45.038048 11875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7104dce7b01a423493a37b6ed694da07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.049379 11878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218ea9985e1d40a48892a8e01aaaef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.058972 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250113 08:57:45.060276 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.061566 11881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7522e543294e28ac66b2faf6001cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.064807 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 218ea9985e1d40a48892a8e01aaaef39 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250113 08:57:45.068359 12026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7104dce7b01a423493a37b6ed694da07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.071892 11950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7104dce7b01a423493a37b6ed694da07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.072474 11956 tablet_service.cc:1467] Processing CreateTablet for tablet 218ea9985e1d40a48892a8e01aaaef39 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250113 08:57:45.075158 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250113 08:57:45.032822 12083 tablet_bootstrap.cc:654] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.022228 12084 consensus_queue.cc:260] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.079846 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218ea9985e1d40a48892a8e01aaaef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.080937 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7522e543294e28ac66b2faf6001cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.083885 12082 tablet_bootstrap.cc:654] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.087097 12084 ts_tablet_manager.cc:1428] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.081s user 0.002s sys 0.005s
I20250113 08:57:45.096999 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 0afcc48db7a94f1bbc8242282f9da2dd (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250113 08:57:45.097756 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250113 08:57:45.098300 11881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0afcc48db7a94f1bbc8242282f9da2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.099047 12023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.098306 11956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218ea9985e1d40a48892a8e01aaaef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.101456 11879 tablet_service.cc:1467] Processing CreateTablet for tablet a105307f831744ff99c190ff6dd0db1a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250113 08:57:45.102759 11879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a105307f831744ff99c190ff6dd0db1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.102658 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:45.109288 11932 leader_election.cc:343] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.114563 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1c553832b94166a0a6089104a74f6c (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250113 08:57:45.116636 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 0afcc48db7a94f1bbc8242282f9da2dd (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250113 08:57:45.120242 11953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.124614 12084 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.124857 11955 tablet_service.cc:1467] Processing CreateTablet for tablet 0afcc48db7a94f1bbc8242282f9da2dd (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250113 08:57:45.126669 11955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0afcc48db7a94f1bbc8242282f9da2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.133585 12093 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.139060 11880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1c553832b94166a0a6089104a74f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.140149 11954 tablet_service.cc:1467] Processing CreateTablet for tablet a105307f831744ff99c190ff6dd0db1a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250113 08:57:45.134083 12093 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.142725 12093 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.146293 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:45.147750 11932 leader_election.cc:343] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.148823 11954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a105307f831744ff99c190ff6dd0db1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.143241 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1c553832b94166a0a6089104a74f6c (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250113 08:57:45.154557 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1c553832b94166a0a6089104a74f6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 08:57:45.157436 11776 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 53423us
I20250113 08:57:45.159264 12084 tablet_bootstrap.cc:654] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.182782 11957 tablet_service.cc:1467] Processing CreateTablet for tablet 07a14a3957e44925ae0925b645d220eb (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250113 08:57:45.184207 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a14a3957e44925ae0925b645d220eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.190199 11882 tablet_service.cc:1467] Processing CreateTablet for tablet 043006a99fa54df3a6ee86ebd18003a7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250113 08:57:45.191844 11882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043006a99fa54df3a6ee86ebd18003a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.194060 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 07a14a3957e44925ae0925b645d220eb (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250113 08:57:45.195389 11878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a14a3957e44925ae0925b645d220eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.189934 11958 tablet_service.cc:1467] Processing CreateTablet for tablet 043006a99fa54df3a6ee86ebd18003a7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250113 08:57:45.205370 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0afcc48db7a94f1bbc8242282f9da2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.206019 11958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043006a99fa54df3a6ee86ebd18003a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.213953 12088 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.214648 12088 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.219470 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:45.221561 11858 leader_election.cc:343] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.217235 12088 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.223402 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 87a2e0ccdce34284b13fa0924326b262 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250113 08:57:45.227146 12029 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1c553832b94166a0a6089104a74f6c (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250113 08:57:45.235312 12083 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.237404 11949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a2e0ccdce34284b13fa0924326b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.249424 12029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1c553832b94166a0a6089104a74f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.266180 11875 tablet_service.cc:1467] Processing CreateTablet for tablet 87a2e0ccdce34284b13fa0924326b262 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250113 08:57:45.270012 11875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a2e0ccdce34284b13fa0924326b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.270345 11950 tablet_service.cc:1467] Processing CreateTablet for tablet 05d6fc5c56554478ae416a917a542304 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250113 08:57:45.272212 12026 tablet_service.cc:1467] Processing CreateTablet for tablet a105307f831744ff99c190ff6dd0db1a (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250113 08:57:45.273542 12026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a105307f831744ff99c190ff6dd0db1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.244822 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 07a14a3957e44925ae0925b645d220eb (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250113 08:57:45.278244 12028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a14a3957e44925ae0925b645d220eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.279590 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9d15ff319f43f0a1dd5ae717393675 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250113 08:57:45.279876 11874 tablet_service.cc:1467] Processing CreateTablet for tablet 637bd1e4126142b481c15befc679b5ac (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250113 08:57:45.280861 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a9d15ff319f43f0a1dd5ae717393675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.281473 11874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637bd1e4126142b481c15befc679b5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.290359 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 05d6fc5c56554478ae416a917a542304 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250113 08:57:45.259728 12084 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.293666 12084 ts_tablet_manager.cc:1397] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.169s user 0.013s sys 0.005s
I20250113 08:57:45.293994 11879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d6fc5c56554478ae416a917a542304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.294659 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 87a2e0ccdce34284b13fa0924326b262 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250113 08:57:45.296454 12084 raft_consensus.cc:357] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.252480 12083 ts_tablet_manager.cc:1397] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.332s user 0.030s sys 0.017s
I20250113 08:57:45.292372 11883 tablet_service.cc:1467] Processing CreateTablet for tablet 89d1890dc82d4bd889d0300c9fed050d (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250113 08:57:45.297641 11950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d6fc5c56554478ae416a917a542304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.256340 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 043006a99fa54df3a6ee86ebd18003a7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250113 08:57:45.299376 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.301726 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9d15ff319f43f0a1dd5ae717393675 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250113 08:57:45.297212 12084 raft_consensus.cc:383] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.302695 12083 raft_consensus.cc:357] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.315903 12083 raft_consensus.cc:383] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.315871 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 209260c370164f719208bd1404eef071 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250113 08:57:45.316352 12083 raft_consensus.cc:738] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.317057 12083 consensus_queue.cc:260] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.322289 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 05d6fc5c56554478ae416a917a542304 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250113 08:57:45.307322 11957 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9d15ff319f43f0a1dd5ae717393675 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250113 08:57:45.308612 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 637bd1e4126142b481c15befc679b5ac (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250113 08:57:45.302556 11956 tablet_service.cc:1467] Processing CreateTablet for tablet 209260c370164f719208bd1404eef071 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250113 08:57:45.314632 12084 raft_consensus.cc:738] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.333032 12084 consensus_queue.cc:260] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.334935 12084 ts_tablet_manager.cc:1428] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250113 08:57:45.322219 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 08aa17121f5b429b8a32299911c7ef76 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
W20250113 08:57:45.322793 11994 tablet.cc:2367] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:57:45.322348 12083 ts_tablet_manager.cc:1428] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.025s user 0.005s sys 0.003s
I20250113 08:57:45.325702 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 209260c370164f719208bd1404eef071 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250113 08:57:45.326876 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.329337 11883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89d1890dc82d4bd889d0300c9fed050d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.338599 11876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08aa17121f5b429b8a32299911c7ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.306185 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 08aa17121f5b429b8a32299911c7ef76 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250113 08:57:45.339926 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 89d1890dc82d4bd889d0300c9fed050d (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20250113 08:57:45.343348 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:45.343104 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 8133c246c9d74113aec7a002df391ef7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250113 08:57:45.345044 11955 tablet_service.cc:1467] Processing CreateTablet for tablet 637bd1e4126142b481c15befc679b5ac (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250113 08:57:45.330610 12023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a2e0ccdce34284b13fa0924326b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.347332 12032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a9d15ff319f43f0a1dd5ae717393675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.348879 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637bd1e4126142b481c15befc679b5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.349514 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d6fc5c56554478ae416a917a542304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.350548 12083 tablet_bootstrap.cc:492] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
W20250113 08:57:45.353648 11856 leader_election.cc:343] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.354106 11856 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:45.355206 12088 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:45.357092 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:57:45.357331 11856 leader_election.cc:343] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.357734 11856 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:45.358927 12088 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:45.361748 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.362367 12084 tablet_bootstrap.cc:492] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.364850 12082 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.365372 12082 ts_tablet_manager.cc:1397] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.452s user 0.047s sys 0.040s
I20250113 08:57:45.368230 12082 raft_consensus.cc:357] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.368963 12082 raft_consensus.cc:383] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.369797 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a9d15ff319f43f0a1dd5ae717393675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.373551 12031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043006a99fa54df3a6ee86ebd18003a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.373888 11956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209260c370164f719208bd1404eef071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.376214 12027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08aa17121f5b429b8a32299911c7ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.378165 11955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637bd1e4126142b481c15befc679b5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.381163 12087 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.371372 11954 tablet_service.cc:1467] Processing CreateTablet for tablet 89d1890dc82d4bd889d0300c9fed050d (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250113 08:57:45.382104 12084 tablet_bootstrap.cc:654] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.381769 12087 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.382634 11954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89d1890dc82d4bd889d0300c9fed050d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.384039 12087 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:45.386188 12024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209260c370164f719208bd1404eef071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.386133 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:45.386394 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 08aa17121f5b429b8a32299911c7ef76 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250113 08:57:45.387615 11953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08aa17121f5b429b8a32299911c7ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.387411 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:45.388615 12007 leader_election.cc:343] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.363688 11881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209260c370164f719208bd1404eef071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.389815 12028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89d1890dc82d4bd889d0300c9fed050d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.390107 11877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8133c246c9d74113aec7a002df391ef7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 08:57:45.409668 11931 leader_election.cc:343] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.410193 11931 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:45.411495 11931 leader_election.cc:343] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.411926 11931 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:45.413002 12093 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:45.415589 12082 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.415582 12093 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:45.416558 12082 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.418704 11893 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:45.426038 12006 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:45.427305 12082 ts_tablet_manager.cc:1428] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.062s user 0.003s sys 0.003s
I20250113 08:57:45.409626 11950 tablet_service.cc:1467] Processing CreateTablet for tablet 8133c246c9d74113aec7a002df391ef7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250113 08:57:45.427229 12087 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:45.428382 12087 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:45.428740 12087 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.435564 12087 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.440023 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:45.441555 12007 leader_election.cc:343] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.442133 12087 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:45.443111 12083 tablet_bootstrap.cc:654] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.450521 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:45.456712 11893 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.464646 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 8133c246c9d74113aec7a002df391ef7 (DEFAULT_TABLE table=test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250113 08:57:45.466360 12025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8133c246c9d74113aec7a002df391ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.485194 11950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8133c246c9d74113aec7a002df391ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:57:45.487775 12082 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.500141 12084 tablet_bootstrap.cc:492] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.500686 12084 ts_tablet_manager.cc:1397] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.139s user 0.007s sys 0.015s
I20250113 08:57:45.519572 12084 raft_consensus.cc:357] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.520253 12084 raft_consensus.cc:383] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.520485 12084 raft_consensus.cc:738] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.521171 12084 consensus_queue.cc:260] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.529127 12082 tablet_bootstrap.cc:654] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.530654 11893 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:45.544224 12083 tablet_bootstrap.cc:492] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.546272 12083 ts_tablet_manager.cc:1397] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.196s user 0.011s sys 0.046s
I20250113 08:57:45.549088 12083 raft_consensus.cc:357] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.549852 12083 raft_consensus.cc:383] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.550153 12083 raft_consensus.cc:738] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.550863 12083 consensus_queue.cc:260] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.545821 12084 ts_tablet_manager.cc:1428] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20250113 08:57:45.545351 12006 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:45.556205 12087 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:45.557303 12083 ts_tablet_manager.cc:1428] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250113 08:57:45.557871 12084 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.558270 12083 tablet_bootstrap.cc:492] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.563619 12087 raft_consensus.cc:695] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:45.564816 12083 tablet_bootstrap.cc:654] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.564625 12087 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.569972 12082 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.570500 12082 ts_tablet_manager.cc:1397] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.008s
I20250113 08:57:45.571219 12083 tablet_bootstrap.cc:492] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.571691 12083 ts_tablet_manager.cc:1397] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250113 08:57:45.571863 12084 tablet_bootstrap.cc:654] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.573276 12082 raft_consensus.cc:357] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.574087 12082 raft_consensus.cc:383] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.574393 12082 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.574393 12083 raft_consensus.cc:357] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.575266 12083 raft_consensus.cc:383] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.575579 12083 raft_consensus.cc:738] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.575397 12082 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.576422 12083 consensus_queue.cc:260] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.579042 12083 ts_tablet_manager.cc:1428] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250113 08:57:45.579957 12083 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.581779 11791 catalog_manager.cc:5526] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:45.585470 12082 ts_tablet_manager.cc:1428] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20250113 08:57:45.586305 12083 tablet_bootstrap.cc:654] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.605607 12082 tablet_bootstrap.cc:492] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.618093 12084 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.618614 12084 ts_tablet_manager.cc:1397] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.061s user 0.025s sys 0.027s
I20250113 08:57:45.619601 12083 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.618109 12082 tablet_bootstrap.cc:654] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.620251 12083 ts_tablet_manager.cc:1397] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.005s
I20250113 08:57:45.621975 12084 raft_consensus.cc:357] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.622895 12084 raft_consensus.cc:383] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.623237 12084 raft_consensus.cc:738] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.623440 12083 raft_consensus.cc:357] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.624285 12083 raft_consensus.cc:383] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.624622 12083 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.625375 12083 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.623980 12084 consensus_queue.cc:260] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.627928 12083 ts_tablet_manager.cc:1428] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20250113 08:57:45.628909 12083 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.628988 12084 ts_tablet_manager.cc:1428] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250113 08:57:45.630455 12084 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.632714 12082 tablet_bootstrap.cc:492] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.633225 12082 ts_tablet_manager.cc:1397] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.000s
I20250113 08:57:45.635787 12083 tablet_bootstrap.cc:654] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.636071 12082 raft_consensus.cc:357] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.636811 12084 tablet_bootstrap.cc:654] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.636933 12082 raft_consensus.cc:383] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.637375 12082 raft_consensus.cc:738] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.638219 12082 consensus_queue.cc:260] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.647341 12082 ts_tablet_manager.cc:1428] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250113 08:57:45.648396 12082 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.654404 12107 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.654918 12107 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.654595 12082 tablet_bootstrap.cc:654] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.657210 12107 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.658502 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.661023 12041 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:45.659766 12083 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.662189 12083 ts_tablet_manager.cc:1397] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.012s
I20250113 08:57:45.662559 12082 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.662334 11931 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:45.663012 12082 ts_tablet_manager.cc:1397] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20250113 08:57:45.663379 12107 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:45.658190 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:45.664338 12107 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:45.664467 11893 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:45.665417 12088 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.664803 12107 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.665809 12082 raft_consensus.cc:357] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.665906 12088 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.666623 12082 raft_consensus.cc:383] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.666946 12082 raft_consensus.cc:738] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.660028 12084 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.667906 12088 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.667567 12082 consensus_queue.cc:260] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.668224 12084 ts_tablet_manager.cc:1397] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.017s
I20250113 08:57:45.669433 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:45.669986 12082 ts_tablet_manager.cc:1428] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250113 08:57:45.670066 11968 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:45.670974 12082 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.671367 11858 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:45.672247 12088 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:45.672714 12088 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:45.673137 12088 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.674662 12107 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.671088 12084 raft_consensus.cc:357] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.676956 12084 raft_consensus.cc:383] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.677258 12084 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.677910 12082 tablet_bootstrap.cc:654] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.677914 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:45.678642 11893 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.679184 12084 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.684132 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.685055 12041 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:45.664860 12083 raft_consensus.cc:357] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.687310 12083 raft_consensus.cc:383] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.687613 12083 raft_consensus.cc:738] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.688391 12083 consensus_queue.cc:260] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.690469 12084 ts_tablet_manager.cc:1428] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.022s user 0.003s sys 0.004s
I20250113 08:57:45.697782 12083 ts_tablet_manager.cc:1428] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.035s user 0.002s sys 0.005s
I20250113 08:57:45.698834 12083 tablet_bootstrap.cc:492] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.701810 11893 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:45.704713 11932 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:45.705418 12093 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:45.702432 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:45.706979 12082 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.707057 12041 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.707540 12082 ts_tablet_manager.cc:1397] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.021s
I20250113 08:57:45.709771 12084 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.710578 12082 raft_consensus.cc:357] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.711645 12082 raft_consensus.cc:383] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.712033 12082 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.712954 12082 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.714259 12123 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.715355 12083 tablet_bootstrap.cc:654] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.714759 12123 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.715875 12041 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:45.701297 12107 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.703362 12088 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.718896 12084 tablet_bootstrap.cc:654] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.719614 12088 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.720400 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:45.721006 11968 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.721495 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:45.722447 12041 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.725628 12123 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.726495 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:45.728206 11858 leader_election.cc:343] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.729300 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.729945 12042 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:45.730335 11968 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:45.731361 11858 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:45.731376 11856 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:45.732628 12123 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:45.733493 12088 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:45.733846 12088 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:45.734181 12088 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.735437 12107 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.735940 12107 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.738221 12107 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.740577 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:45.741060 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.741521 11893 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:45.741990 12042 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:45.742353 12107 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.742846 12107 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.743075 11932 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:45.744843 12107 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.747167 12107 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:45.747581 12107 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:45.747962 12107 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.748448 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.749346 12042 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:45.750946 11931 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:45.752458 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:45.754472 11932 leader_election.cc:343] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.759205 12082 ts_tablet_manager.cc:1428] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.051s user 0.012s sys 0.029s
I20250113 08:57:45.760211 12082 tablet_bootstrap.cc:492] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.756561 12123 raft_consensus.cc:695] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:45.762607 12088 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.763185 12123 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.764439 12125 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:45.764940 12125 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:45.765350 12125 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.766824 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:45.764463 12041 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:45.766695 12107 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.757706 12093 raft_consensus.cc:695] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:57:45.768774 11858 leader_election.cc:343] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.769851 12088 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.769647 12093 consensus_queue.cc:237] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.770946 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:45.771473 12107 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.771562 12042 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.772799 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:45.773475 12040 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.779891 12042 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:45.780570 12040 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:45.772294 12083 tablet_bootstrap.cc:492] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.781289 11856 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:45.782028 11931 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:45.771705 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:45.782744 11893 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.782887 12093 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:45.783517 12093 raft_consensus.cc:695] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:45.782848 12125 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.783942 11789 catalog_manager.cc:5526] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:45.784555 12093 consensus_queue.cc:237] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.781306 12083 ts_tablet_manager.cc:1397] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.011s
I20250113 08:57:45.786929 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:45.788082 12083 raft_consensus.cc:357] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:45.788473 11932 leader_election.cc:343] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:45.788775 12083 raft_consensus.cc:383] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.789584 12107 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.790035 12107 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.790750 12125 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.789052 12083 raft_consensus.cc:738] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.792194 12107 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:45.792052 12083 consensus_queue.cc:260] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.793728 12088 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:45.793933 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:45.794236 12088 raft_consensus.cc:695] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:45.794596 12083 ts_tablet_manager.cc:1428] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250113 08:57:45.794745 11892 raft_consensus.cc:2388] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:45.790566 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:45.795451 12040 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.796967 11893 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:45.797501 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:45.795068 12088 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.798717 11931 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:45.799862 12125 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.782256 12082 tablet_bootstrap.cc:654] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.804466 12083 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.807281 12125 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:45.791033 12084 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.809887 12084 ts_tablet_manager.cc:1397] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.100s user 0.015s sys 0.007s
I20250113 08:57:45.815275 12083 tablet_bootstrap.cc:654] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.816243 12084 raft_consensus.cc:357] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.817463 12084 raft_consensus.cc:383] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.817952 12084 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.818787 12084 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.822170 12084 ts_tablet_manager.cc:1428] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.012s user 0.002s sys 0.007s
I20250113 08:57:45.823458 12084 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.831888 11789 catalog_manager.cc:5526] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:45.828716 11790 catalog_manager.cc:5526] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:45.834928 12084 tablet_bootstrap.cc:654] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.839123 12040 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:45.845161 12082 tablet_bootstrap.cc:492] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.846959 12082 ts_tablet_manager.cc:1397] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.016s
I20250113 08:57:45.847493 11931 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:45.848671 12093 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:45.849140 12093 raft_consensus.cc:695] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:45.849506 12082 raft_consensus.cc:357] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.850216 12082 raft_consensus.cc:383] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.849929 12093 consensus_queue.cc:237] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.850471 12082 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.851073 12082 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.853462 12082 ts_tablet_manager.cc:1428] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250113 08:57:45.854580 12082 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.856427 11791 catalog_manager.cc:5526] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:45.861110 12082 tablet_bootstrap.cc:654] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.871635 12083 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.872231 12083 ts_tablet_manager.cc:1397] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.068s user 0.016s sys 0.001s
I20250113 08:57:45.871825 11788 catalog_manager.cc:5526] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:45.874991 12083 raft_consensus.cc:357] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.876057 12083 raft_consensus.cc:383] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.876463 12083 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.877499 12083 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.882742 12083 ts_tablet_manager.cc:1428] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250113 08:57:45.886283 12083 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.885099 12084 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.886912 12084 ts_tablet_manager.cc:1397] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.064s user 0.030s sys 0.019s
I20250113 08:57:45.888875 12082 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.889392 12082 ts_tablet_manager.cc:1397] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.008s
I20250113 08:57:45.889323 12084 raft_consensus.cc:357] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.890131 12084 raft_consensus.cc:383] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.890410 12084 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.891153 12084 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.891810 12083 tablet_bootstrap.cc:654] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.891925 12082 raft_consensus.cc:357] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.892604 12082 raft_consensus.cc:383] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.892892 12082 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.893707 12084 ts_tablet_manager.cc:1428] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250113 08:57:45.893594 12082 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.894630 12084 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.900604 12084 tablet_bootstrap.cc:654] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.906347 12082 ts_tablet_manager.cc:1428] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250113 08:57:45.917451 12082 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.919467 12083 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.919467 12084 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.920150 12083 ts_tablet_manager.cc:1397] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.009s
I20250113 08:57:45.920150 12084 ts_tablet_manager.cc:1397] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.013s
I20250113 08:57:45.922026 12082 tablet_bootstrap.cc:654] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.922911 12083 raft_consensus.cc:357] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.923048 12084 raft_consensus.cc:357] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.923722 12083 raft_consensus.cc:383] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.923774 12084 raft_consensus.cc:383] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.924103 12083 raft_consensus.cc:738] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.924172 12084 raft_consensus.cc:738] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.925068 12083 consensus_queue.cc:260] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.925030 12084 consensus_queue.cc:260] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.926971 12082 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.927178 12084 ts_tablet_manager.cc:1428] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250113 08:57:45.927400 12082 ts_tablet_manager.cc:1397] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20250113 08:57:45.927178 12083 ts_tablet_manager.cc:1428] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:57:45.928964 12083 tablet_bootstrap.cc:492] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.930275 12084 tablet_bootstrap.cc:492] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.929967 12082 raft_consensus.cc:357] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.930722 12082 raft_consensus.cc:383] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.931010 12082 raft_consensus.cc:738] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.931732 12082 consensus_queue.cc:260] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.936172 12083 tablet_bootstrap.cc:654] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.937098 12084 tablet_bootstrap.cc:654] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.937933 12082 ts_tablet_manager.cc:1428] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250113 08:57:45.938841 12082 tablet_bootstrap.cc:492] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.944995 12082 tablet_bootstrap.cc:654] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.950028 12084 tablet_bootstrap.cc:492] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:45.950033 12083 tablet_bootstrap.cc:492] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:45.950598 12084 ts_tablet_manager.cc:1397] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20250113 08:57:45.950675 12083 ts_tablet_manager.cc:1397] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20250113 08:57:45.958882 12084 raft_consensus.cc:357] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.959502 12082 tablet_bootstrap.cc:492] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:45.959662 12084 raft_consensus.cc:383] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.959228 12083 raft_consensus.cc:357] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.959976 12082 ts_tablet_manager.cc:1397] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.008s
I20250113 08:57:45.960242 12083 raft_consensus.cc:383] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.960582 12083 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.959998 12084 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.961336 12083 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.962570 12087 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.962503 12082 raft_consensus.cc:357] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.962635 12084 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.963291 12082 raft_consensus.cc:383] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:45.963161 12087 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.963574 12082 raft_consensus.cc:738] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:45.964954 12084 ts_tablet_manager.cc:1428] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.014s user 0.003s sys 0.008s
I20250113 08:57:45.965157 12087 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:45.965893 12084 tablet_bootstrap.cc:492] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:45.966975 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:45.967658 11893 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:45.968024 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:45.968647 11968 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:45.968902 12006 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:45.970180 12087 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:45.970999 12087 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:45.971487 12087 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.972067 12083 ts_tablet_manager.cc:1428] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.021s user 0.009s sys 0.003s
I20250113 08:57:45.964197 12082 consensus_queue.cc:260] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.973354 12084 tablet_bootstrap.cc:654] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:45.978575 12083 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:45.983287 12110 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:45.984409 12082 ts_tablet_manager.cc:1428] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250113 08:57:45.984073 12110 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.990052 12087 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:45.993842 12082 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:45.994050 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:45.993935 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:45.994828 11892 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:45.995081 11893 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:45.997848 12006 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:45.998886 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:46.001192 12007 leader_election.cc:343] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.003185 12110 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:46.003597 12082 tablet_bootstrap.cc:654] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.004320 12110 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.005237 12110 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.017580 12082 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:46.018193 12082 ts_tablet_manager.cc:1397] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20250113 08:57:46.019764 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:46.020502 11968 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.021238 12082 raft_consensus.cc:357] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.022235 12082 raft_consensus.cc:383] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.022500 12082 raft_consensus.cc:738] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.023118 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:46.023288 12082 consensus_queue.cc:260] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.024178 11967 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:46.026397 12007 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:46.027810 12110 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:46.028699 12110 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:46.029302 12110 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:46.029803 12087 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:46.029887 12110 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.030576 12087 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:46.031016 12087 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:46.031487 12087 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.035387 11892 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:46.036747 12082 ts_tablet_manager.cc:1428] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250113 08:57:46.037159 12083 tablet_bootstrap.cc:654] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.037814 12088 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.038398 12088 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.041615 12082 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:46.042881 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:46.043862 11967 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:46.045722 11858 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:46.047384 12123 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:46.047858 12123 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:46.048262 12123 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.037783 11968 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:46.055236 12007 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:57:46.060480 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 62731.6us
I20250113 08:57:46.066416 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.067423 12040 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:46.087137 12146 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:46.087697 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:46.088546 12110 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.091341 12128 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.091590 12110 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:46.091975 12128 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.094805 12128 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.097043 12084 tablet_bootstrap.cc:492] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:46.098680 12084 ts_tablet_manager.cc:1397] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.133s user 0.044s sys 0.062s
I20250113 08:57:46.100019 12082 tablet_bootstrap.cc:654] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.100294 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:46.102193 12007 leader_election.cc:343] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.102857 12084 raft_consensus.cc:357] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.103873 12084 raft_consensus.cc:383] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.104343 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.104920 12088 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.104869 12150 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.105298 12040 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:46.104234 12084 raft_consensus.cc:738] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.106734 12084 consensus_queue.cc:260] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.107003 11856 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:46.103905 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:46.108783 12084 ts_tablet_manager.cc:1428] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250113 08:57:46.075484 12123 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.109189 12146 raft_consensus.cc:695] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:57:46.110028 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 47845.2us
I20250113 08:57:46.110306 12146 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.111430 12123 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.111788 12083 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:46.112782 12083 ts_tablet_manager.cc:1397] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.140s user 0.012s sys 0.022s
I20250113 08:57:46.112970 12084 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:46.112145 12123 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:46.112813 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:46.108500 11968 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:46.116847 11966 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.117067 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:46.116245 12123 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:46.118288 11892 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.121462 12082 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:46.122017 12082 ts_tablet_manager.cc:1397] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.031s
I20250113 08:57:46.121917 12083 raft_consensus.cc:357] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.122681 12083 raft_consensus.cc:383] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.122874 12083 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.120847 12123 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.120959 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:46.123899 12083 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.124419 12040 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.126314 12083 ts_tablet_manager.cc:1428] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.013s user 0.001s sys 0.005s
I20250113 08:57:46.127213 12083 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:46.131573 12123 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:46.117635 12006 leader_election.cc:343] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.136310 12088 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.136888 12088 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.137715 12082 raft_consensus.cc:357] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.138409 12082 raft_consensus.cc:383] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.138661 12082 raft_consensus.cc:738] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.139402 12088 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.139385 12082 consensus_queue.cc:260] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.143164 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:46.144773 11858 leader_election.cc:343] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.150732 12150 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.151191 11892 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:46.152133 12040 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:46.134860 12123 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.154016 12082 ts_tablet_manager.cc:1428] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20250113 08:57:46.154650 12006 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:46.152695 12093 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.157981 12082 tablet_bootstrap.cc:492] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:46.157508 12093 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.240289 11788 catalog_manager.cc:5526] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:46.242127 11966 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:46.243454 11858 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:46.192533 12125 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20250113 08:57:46.246842 12083 tablet_bootstrap.cc:654] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.248936 12128 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:46.169514 12146 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:46.249542 12128 raft_consensus.cc:695] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:46.249819 12146 raft_consensus.cc:695] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:46.250654 12128 consensus_queue.cc:237] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.254936 11966 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.256907 12107 consensus_queue.cc:1035] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.251060 12146 consensus_queue.cc:237] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.186882 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:46.260468 12040 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.264853 12084 tablet_bootstrap.cc:654] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.168582 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:46.182109 12088 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.224462 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.268704 11968 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.179286 12087 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.298307 12128 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250113 08:57:46.300192 11856 leader_election.cc:343] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.300802 11856 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:46.307126 12093 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.269428 12088 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.313421 12128 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:46.311141 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.315618 12083 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:46.316080 12083 ts_tablet_manager.cc:1397] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.189s user 0.011s sys 0.010s
I20250113 08:57:46.311195 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:46.330258 11891 raft_consensus.cc:2388] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:46.340163 12087 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:46.334005 12088 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.357520 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.363191 12123 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.363965 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:46.364020 12123 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.350205 12128 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250113 08:57:46.370203 12123 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.397748 12087 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:46.419327 12006 leader_election.cc:343] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:46.419579 11856 leader_election.cc:343] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:46.472666 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:46.523617 12083 raft_consensus.cc:357] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.524158 12083 raft_consensus.cc:383] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.524386 12083 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.524873 12083 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.517786 12088 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.526453 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:46.526667 12088 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.527122 11964 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.440654 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:46.485564 11968 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:46.476050 11893 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.463303 12040 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:46.434283 12087 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:46.520691 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 104743us
W20250113 08:57:46.521682 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 103944us
I20250113 08:57:46.539892 12083 ts_tablet_manager.cc:1428] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.224s user 0.010s sys 0.005s
I20250113 08:57:46.538144 12125 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.540963 12083 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:46.540518 12125 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.541278 12087 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:46.542851 12125 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.529018 11788 catalog_manager.cc:5526] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:46.544613 12088 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.526001 12107 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250113 08:57:46.547277 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:46.547998 12158 log.cc:926] Time spent T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20250113 08:57:46.547870 11856 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:46.552973 12082 tablet_bootstrap.cc:654] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.537346 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.558243 12088 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.560796 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:46.538848 11932 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:46.570787 12123 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:46.574730 12123 raft_consensus.cc:695] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:46.557148 11964 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:46.577912 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:46.578367 11858 leader_election.cc:343] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.579254 11858 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:46.578310 12007 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:46.581707 12125 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.582434 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.583878 12126 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:46.585572 12128 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250113 08:57:46.585520 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 53147.4us
W20250113 08:57:46.587575 12007 leader_election.cc:343] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:46.585430 11856 leader_election.cc:343] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.576565 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:46.598453 11858 leader_election.cc:343] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.599545 11858 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:46.602175 12128 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:46.602962 11791 catalog_manager.cc:5526] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
W20250113 08:57:46.592150 12006 leader_election.cc:343] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.592008 12164 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250113 08:57:46.585915 12042 raft_consensus.cc:2388] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:57:46.586030 12087 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:46.580549 12123 consensus_queue.cc:237] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.626293 12087 raft_consensus.cc:695] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:46.636492 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:46.610283 12006 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:46.640055 12110 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:46.639401 12083 tablet_bootstrap.cc:654] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:46.640249 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 48188.5us
I20250113 08:57:46.641362 12084 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:46.641731 12084 ts_tablet_manager.cc:1397] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.529s user 0.023s sys 0.001s
I20250113 08:57:46.644254 12110 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.646231 12084 raft_consensus.cc:357] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.647010 12084 raft_consensus.cc:383] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.647279 12084 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.648036 12125 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:46.647887 12084 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.648986 12087 consensus_queue.cc:237] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.650269 12084 ts_tablet_manager.cc:1428] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250113 08:57:46.654242 12084 tablet_bootstrap.cc:492] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:46.658430 11856 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:46.662281 12146 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.662884 12146 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.670923 12084 tablet_bootstrap.cc:654] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.671033 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:46.666080 12146 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:57:46.685444 12006 leader_election.cc:343] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.674593 12110 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.675774 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:46.692200 12082 tablet_bootstrap.cc:492] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:46.692677 12082 ts_tablet_manager.cc:1397] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.535s user 0.020s sys 0.011s
I20250113 08:57:46.697849 12082 raft_consensus.cc:357] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.698860 12082 raft_consensus.cc:383] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.699199 12082 raft_consensus.cc:738] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.699921 12082 consensus_queue.cc:260] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:46.705266 12007 leader_election.cc:343] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.705922 12007 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:46.707165 12110 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:46.708575 12082 ts_tablet_manager.cc:1428] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.016s user 0.004s sys 0.004s
W20250113 08:57:46.689790 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 334436us
I20250113 08:57:46.709769 12082 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:46.745158 12088 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:46.746510 12128 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.748696 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.775121 12107 consensus_queue.cc:1035] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.777571 12107 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.779194 11931 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:46.781407 12168 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:46.780165 11790 catalog_manager.cc:5526] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:46.796712 12083 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:46.797216 12083 ts_tablet_manager.cc:1397] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.256s user 0.008s sys 0.017s
W20250113 08:57:46.797641 12173 log.cc:926] Time spent T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.120s user 0.000s sys 0.002s
I20250113 08:57:46.800940 12083 raft_consensus.cc:357] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.802559 12083 raft_consensus.cc:383] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.802930 12083 raft_consensus.cc:738] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.804163 12083 consensus_queue.cc:260] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.811417 12083 ts_tablet_manager.cc:1428] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20250113 08:57:46.812594 12123 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:46.817370 12128 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:46.820142 12083 tablet_bootstrap.cc:492] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:46.854687 12084 tablet_bootstrap.cc:492] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:46.855333 12084 ts_tablet_manager.cc:1397] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.204s user 0.017s sys 0.006s
I20250113 08:57:46.833703 12123 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.849864 12168 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:46.858606 12179 log.cc:926] Time spent T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20250113 08:57:46.858294 12168 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.871894 12168 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.876262 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.876498 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:46.877215 12037 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
W20250113 08:57:46.878872 11932 leader_election.cc:343] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:46.880579 11931 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:46.881114 12123 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:46.884622 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:46.886873 12168 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:46.887573 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:46.887670 12168 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:46.888617 11967 raft_consensus.cc:2388] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:46.889617 11791 catalog_manager.cc:5526] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:46.891163 12082 tablet_bootstrap.cc:654] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.891964 12150 consensus_queue.cc:1035] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250113 08:57:46.894865 11858 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:46.888496 12168 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.897516 12123 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:46.901082 12110 consensus_queue.cc:1035] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20250113 08:57:46.942888 12084 raft_consensus.cc:357] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.947021 12084 raft_consensus.cc:383] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:46.947429 12084 raft_consensus.cc:738] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:46.948475 12084 consensus_queue.cc:260] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.955482 12084 ts_tablet_manager.cc:1428] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.100s user 0.002s sys 0.005s
I20250113 08:57:46.963469 12084 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:46.975965 12083 tablet_bootstrap.cc:654] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.983387 12168 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:46.986856 12084 tablet_bootstrap.cc:654] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:46.993726 12123 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:47.012506 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.017103 12082 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:47.017560 12082 ts_tablet_manager.cc:1397] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.308s user 0.013s sys 0.027s
W20250113 08:57:47.016979 11932 leader_election.cc:343] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.019131 12168 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.021791 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:47.022570 12038 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.024219 12082 raft_consensus.cc:357] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.026087 12082 raft_consensus.cc:383] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.026307 12082 raft_consensus.cc:738] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.026883 12082 consensus_queue.cc:260] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.035533 12082 ts_tablet_manager.cc:1428] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20250113 08:57:47.039827 12082 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:47.051678 12082 tablet_bootstrap.cc:654] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.038152 12038 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:47.066905 11931 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:47.068079 12168 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.068703 12168 raft_consensus.cc:695] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:47.070029 12083 tablet_bootstrap.cc:492] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.069730 12168 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.070470 12083 ts_tablet_manager.cc:1397] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.251s user 0.019s sys 0.027s
I20250113 08:57:47.074134 12123 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.074806 12123 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.077937 12123 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.082342 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:47.081161 12083 raft_consensus.cc:357] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.110939 12083 raft_consensus.cc:383] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.111320 12083 raft_consensus.cc:738] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.111999 12083 consensus_queue.cc:260] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:47.112303 11856 leader_election.cc:343] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.114506 12083 ts_tablet_manager.cc:1428] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.044s user 0.007s sys 0.002s
I20250113 08:57:47.083457 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.116532 12083 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.116597 12146 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.116950 12082 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:47.117453 12082 ts_tablet_manager.cc:1397] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.078s user 0.024s sys 0.009s
I20250113 08:57:47.117170 12146 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.118711 12084 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:47.119140 12084 ts_tablet_manager.cc:1397] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.163s user 0.022s sys 0.016s
I20250113 08:57:47.120347 12146 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.121882 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.122268 12084 raft_consensus.cc:357] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.123191 12084 raft_consensus.cc:383] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 08:57:47.123481 11858 leader_election.cc:343] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.124204 11858 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:47.124449 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.123574 12084 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.125284 11890 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:47.125582 12176 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:47.126034 12084 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.128597 12084 ts_tablet_manager.cc:1428] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20250113 08:57:47.129515 12084 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:47.132835 11789 catalog_manager.cc:5526] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:47.141180 12006 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:47.134382 12082 raft_consensus.cc:357] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.143399 12082 raft_consensus.cc:383] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.143685 12082 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.095666 12125 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.144657 12082 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.145367 12146 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.096772 12123 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20250113 08:57:47.155882 12082 ts_tablet_manager.cc:1428] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.038s user 0.006s sys 0.003s
I20250113 08:57:47.156824 12082 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:47.158834 12110 consensus_queue.cc:1035] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.159708 12083 tablet_bootstrap.cc:654] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.150292 12146 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.160741 12146 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.151245 12125 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.164009 11967 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:47.174396 12125 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.175336 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.176147 11890 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:47.176314 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:47.178028 11932 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:47.179661 12125 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.181742 12125 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.191107 12084 tablet_bootstrap.cc:654] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.191248 12110 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.192483 12082 tablet_bootstrap.cc:654] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.187297 12125 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.192436 12110 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.195811 12146 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.196573 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.207038 11890 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:47.199337 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.200258 12110 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.209231 12083 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.209201 12038 raft_consensus.cc:2388] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:47.212481 12083 ts_tablet_manager.cc:1397] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.096s user 0.012s sys 0.025s
I20250113 08:57:47.217686 12146 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.222213 12006 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:47.227491 12007 leader_election.cc:343] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.227759 12150 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.228266 12150 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.228672 12150 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.228931 12123 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.223822 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.225003 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.237093 12087 consensus_queue.cc:1035] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.239503 11890 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.252135 12083 raft_consensus.cc:357] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.252913 12083 raft_consensus.cc:383] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.253182 12083 raft_consensus.cc:738] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.253850 12083 consensus_queue.cc:260] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.258431 12083 ts_tablet_manager.cc:1428] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.046s user 0.002s sys 0.006s
I20250113 08:57:47.259351 12083 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.259519 12082 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:47.259987 12082 ts_tablet_manager.cc:1397] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.009s
I20250113 08:57:47.269493 12150 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.272248 12150 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.273841 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.274646 11893 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.278357 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.278649 12082 raft_consensus.cc:357] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.279378 12082 raft_consensus.cc:383] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.279649 12082 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
W20250113 08:57:47.280360 12007 leader_election.cc:343] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.282691 12125 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.284662 11967 raft_consensus.cc:2388] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 1: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:57:47.286507 12082 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.286826 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.288852 12082 ts_tablet_manager.cc:1428] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20250113 08:57:47.290221 11890 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:47.294170 12125 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.300158 12083 tablet_bootstrap.cc:654] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.295552 12082 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:47.292879 12006 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.301451 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:47.303988 12199 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.304607 12199 raft_consensus.cc:695] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:47.306015 11893 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:47.305682 12199 consensus_queue.cc:237] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.314961 12006 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.317903 12146 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.318713 12146 raft_consensus.cc:695] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:47.320133 12146 consensus_queue.cc:237] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.328728 12084 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:47.329206 12084 ts_tablet_manager.cc:1397] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.200s user 0.007s sys 0.029s
I20250113 08:57:47.340533 12084 raft_consensus.cc:357] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.341305 12084 raft_consensus.cc:383] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.341591 12084 raft_consensus.cc:738] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.343478 12084 consensus_queue.cc:260] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.355540 12093 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.356092 12093 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.366659 12093 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.371976 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.373131 11893 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:47.373623 11931 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:47.374167 12084 ts_tablet_manager.cc:1428] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.045s user 0.008s sys 0.002s
I20250113 08:57:47.365833 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:47.378890 11931 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:47.377104 12093 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 08:57:47.379987 12125 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.383188 12083 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.383795 12083 ts_tablet_manager.cc:1397] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.125s user 0.024s sys 0.031s
I20250113 08:57:47.385872 12084 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:47.387327 12083 raft_consensus.cc:357] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.388126 12083 raft_consensus.cc:383] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.388448 12083 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.389554 12082 tablet_bootstrap.cc:654] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.389151 12083 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.392082 12083 ts_tablet_manager.cc:1428] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20250113 08:57:47.393092 12083 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.397910 12084 tablet_bootstrap.cc:654] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.404173 12083 tablet_bootstrap.cc:654] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.411381 12125 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:47.418380 11791 catalog_manager.cc:5526] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:47.420614 11791 catalog_manager.cc:5526] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:47.437268 12082 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:47.437961 12082 ts_tablet_manager.cc:1397] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.143s user 0.013s sys 0.031s
I20250113 08:57:47.444164 12082 raft_consensus.cc:357] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.445202 12082 raft_consensus.cc:383] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.445587 12082 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.446419 12082 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.452185 12082 ts_tablet_manager.cc:1428] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20250113 08:57:47.457767 12082 tablet_bootstrap.cc:492] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:47.458734 12084 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:47.460675 12084 ts_tablet_manager.cc:1397] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.017s
I20250113 08:57:47.461130 12083 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.461582 12083 ts_tablet_manager.cc:1397] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.069s user 0.024s sys 0.035s
I20250113 08:57:47.461349 12128 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.462240 12128 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.464979 12083 raft_consensus.cc:357] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.465742 12083 raft_consensus.cc:383] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.465765 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.466101 12083 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.466876 12083 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:47.467536 11858 leader_election.cc:343] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.469226 12083 ts_tablet_manager.cc:1428] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:57:47.470237 12083 tablet_bootstrap.cc:492] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.470106 12128 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.476663 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:47.477557 12038 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:47.479882 12083 tablet_bootstrap.cc:654] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.481431 11856 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.484476 12128 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.485071 12128 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.485599 12128 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.487085 11893 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.492728 12083 tablet_bootstrap.cc:492] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.493184 12083 ts_tablet_manager.cc:1397] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.016s
I20250113 08:57:47.491228 12082 tablet_bootstrap.cc:654] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.496160 12083 raft_consensus.cc:357] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.496889 12083 raft_consensus.cc:383] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.497160 12083 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.497877 12083 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.500599 12083 ts_tablet_manager.cc:1428] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:57:47.501796 12083 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.510139 12083 tablet_bootstrap.cc:654] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.512851 12084 raft_consensus.cc:357] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.517354 12084 raft_consensus.cc:383] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.517840 12084 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.516386 12125 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.519049 12084 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.516156 12128 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.523700 12083 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.524209 12083 ts_tablet_manager.cc:1397] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.008s
I20250113 08:57:47.526177 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.527155 12083 raft_consensus.cc:357] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.527863 12083 raft_consensus.cc:383] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 08:57:47.527945 11858 leader_election.cc:343] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.528159 12083 raft_consensus.cc:738] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.528949 12083 consensus_queue.cc:260] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.529625 12125 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.530210 12125 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.531741 12083 ts_tablet_manager.cc:1428] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250113 08:57:47.532658 12083 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.532814 12125 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.533732 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.535220 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:57:47.539052 11931 leader_election.cc:343] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.539196 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:47.539907 12038 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:47.539880 11932 leader_election.cc:343] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.538341 12128 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.543296 11932 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:47.551752 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:47.552271 12125 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:47.547318 12084 ts_tablet_manager.cc:1428] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.086s user 0.000s sys 0.011s
I20250113 08:57:47.553869 12084 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:47.555487 12083 tablet_bootstrap.cc:654] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.550768 12038 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:47.560602 11856 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.562129 12128 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.563231 12128 raft_consensus.cc:695] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:47.564343 12128 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.570714 12082 tablet_bootstrap.cc:492] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:47.571264 12082 ts_tablet_manager.cc:1397] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.118s user 0.021s sys 0.009s
I20250113 08:57:47.576122 12082 raft_consensus.cc:357] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.577004 12082 raft_consensus.cc:383] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.577338 12082 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.579027 12125 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.589237 12082 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.591605 12084 tablet_bootstrap.cc:654] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.597973 12082 ts_tablet_manager.cc:1428] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.026s user 0.009s sys 0.002s
I20250113 08:57:47.607169 11790 catalog_manager.cc:5526] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:47.615047 12082 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:47.631860 12082 tablet_bootstrap.cc:654] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.613863 12083 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.636900 12083 ts_tablet_manager.cc:1397] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.009s
I20250113 08:57:47.640187 12083 raft_consensus.cc:357] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.641312 12083 raft_consensus.cc:383] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.641651 12083 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.644088 12083 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.649333 12146 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.653123 12083 ts_tablet_manager.cc:1428] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250113 08:57:47.653061 12146 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.654337 12083 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.655987 12146 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.659607 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:47.661337 12007 leader_election.cc:343] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.662633 12176 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.663360 12176 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.674983 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.675936 11893 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:47.677631 12006 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.679659 12176 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.680297 12146 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.682554 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.684568 11963 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:47.682914 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:47.685904 12038 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:47.686178 11858 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:47.683157 12146 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.687170 12146 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.687283 12176 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.687836 12176 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.688248 12176 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.698431 12084 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:47.698961 12084 ts_tablet_manager.cc:1397] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.145s user 0.012s sys 0.008s
I20250113 08:57:47.707413 12084 raft_consensus.cc:357] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.708178 12084 raft_consensus.cc:383] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.708456 12084 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.709082 12084 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.711237 12084 ts_tablet_manager.cc:1428] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250113 08:57:47.712915 12146 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.713806 12084 tablet_bootstrap.cc:492] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:47.708528 12176 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.715306 12146 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.716228 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.716519 12176 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.716702 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.717046 11893 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:47.718741 12007 leader_election.cc:343] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.719586 12083 tablet_bootstrap.cc:654] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.720543 12082 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:47.721017 12082 ts_tablet_manager.cc:1397] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.031s
I20250113 08:57:47.721451 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:47.722241 12038 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.724370 12082 raft_consensus.cc:357] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.728470 12082 raft_consensus.cc:383] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.728742 12082 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.729565 12082 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.732077 12082 ts_tablet_manager.cc:1428] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250113 08:57:47.733812 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.734613 11967 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.744495 12083 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.739854 12038 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:47.758028 12146 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250113 08:57:47.761277 12083 ts_tablet_manager.cc:1397] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.107s user 0.020s sys 0.028s
I20250113 08:57:47.761615 11856 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:47.762274 11893 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:47.762888 12176 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.763329 12176 raft_consensus.cc:695] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:47.764118 12176 consensus_queue.cc:237] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.764115 12006 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.766111 12110 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.766566 12110 raft_consensus.cc:695] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:47.767270 12199 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.767395 12110 consensus_queue.cc:237] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.746544 12150 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.771320 12150 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.773322 12082 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:47.777765 11967 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:47.773551 12083 raft_consensus.cc:357] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.780231 12083 raft_consensus.cc:383] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.780608 12083 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.781749 12083 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.784749 12083 ts_tablet_manager.cc:1428] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20250113 08:57:47.785529 12083 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.800745 12082 tablet_bootstrap.cc:654] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.808775 11791 catalog_manager.cc:5526] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:47.810217 11790 catalog_manager.cc:5526] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:47.816454 12150 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.825551 12084 tablet_bootstrap.cc:654] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.824877 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.826587 11890 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:47.825241 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.829015 12006 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:47.831554 12007 leader_election.cc:343] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.833509 12110 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.834154 12110 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.835254 12110 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.847415 12083 tablet_bootstrap.cc:654] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.857582 12146 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.848652 12150 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.859956 12150 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.872022 12150 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.874023 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:47.877627 12006 leader_election.cc:343] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.880164 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.880952 11963 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:47.883073 12007 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:47.884534 12222 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:47.886346 12150 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.886842 12150 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.887296 12150 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.890133 12110 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.893723 12084 tablet_bootstrap.cc:492] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:47.902087 12082 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:47.902487 12084 ts_tablet_manager.cc:1397] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.189s user 0.013s sys 0.024s
I20250113 08:57:47.902612 12082 ts_tablet_manager.cc:1397] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.130s user 0.026s sys 0.014s
I20250113 08:57:47.894174 12222 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.906711 12082 raft_consensus.cc:357] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.907578 12082 raft_consensus.cc:383] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.907091 12110 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.907903 12082 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.908660 12082 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.910490 12150 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.911758 12082 ts_tablet_manager.cc:1428] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250113 08:57:47.904100 12222 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.912195 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.912427 12150 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.912758 11893 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.912993 12082 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:47.913867 12222 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.913575 12084 raft_consensus.cc:357] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.914425 12084 raft_consensus.cc:383] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.914711 12084 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.915344 12084 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.918885 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.919080 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.919486 11966 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.921572 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.923461 11891 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:47.923712 12007 leader_election.cc:343] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.924219 12084 ts_tablet_manager.cc:1428] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.021s user 0.002s sys 0.005s
I20250113 08:57:47.925127 12084 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:47.926215 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.929754 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.930436 11890 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
W20250113 08:57:47.930670 12007 leader_election.cc:343] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.931950 12006 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.932823 12222 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:47.933490 12222 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:47.936133 12222 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.936831 11893 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:47.938272 12006 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:47.940768 12150 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.942160 12150 raft_consensus.cc:695] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:47.943472 12083 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:47.943058 12150 consensus_queue.cc:237] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.943998 12083 ts_tablet_manager.cc:1397] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.159s user 0.016s sys 0.009s
I20250113 08:57:47.943696 11966 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:47.947050 12083 raft_consensus.cc:357] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.948009 12083 raft_consensus.cc:383] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:47.948330 12083 raft_consensus.cc:738] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:47.949120 12083 consensus_queue.cc:260] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.952387 12007 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:47.953315 12226 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:47.953778 12226 raft_consensus.cc:695] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:47.954540 12226 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.957988 12222 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.959205 11891 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:47.961753 12083 ts_tablet_manager.cc:1428] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250113 08:57:47.962500 12083 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:47.967525 12222 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.968322 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:47.968746 12082 tablet_bootstrap.cc:654] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.968958 11891 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:47.972774 12222 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.973372 12222 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.975663 12083 tablet_bootstrap.cc:654] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.976315 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:47.977725 12006 leader_election.cc:343] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.977874 12084 tablet_bootstrap.cc:654] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:47.968339 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:47.984840 12125 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.985330 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:47.985598 11891 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:47.985416 12125 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.986649 12150 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:47.986999 12006 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:47.987216 12150 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:47.991236 12150 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:47.991531 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:47.992640 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:47.993151 11931 leader_election.cc:343] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.994369 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:47.995050 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:47.995823 11932 leader_election.cc:343] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.996024 11890 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:47.996573 11932 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:47.997603 12006 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:47.997998 12168 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:47.998674 12007 leader_election.cc:343] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:47.999006 12125 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:47.994696 11789 catalog_manager.cc:5526] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:47.994328 12222 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:48.000504 11789 catalog_manager.cc:5526] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:48.003288 12226 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
W20250113 08:57:48.004290 12007 leader_election.cc:343] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.004895 12222 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:48.005398 12007 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.006757 12222 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:48.007203 12150 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.004742 12226 raft_consensus.cc:695] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:57:48.007596 12007 leader_election.cc:343] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.008519 12226 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.007282 12222 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.019552 12082 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:48.020164 12082 ts_tablet_manager.cc:1397] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.003s
I20250113 08:57:48.021224 12083 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:48.022703 12083 ts_tablet_manager.cc:1397] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.010s
I20250113 08:57:48.025444 12125 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.023612 12222 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.026098 12125 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.027823 12083 raft_consensus.cc:357] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.028592 12083 raft_consensus.cc:383] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.029311 12222 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:48.029448 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:48.029557 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:48.029557 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.030076 11891 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:48.031168 11932 leader_election.cc:343] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:48.031258 12007 leader_election.cc:343] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.028916 12083 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.032559 12083 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.032850 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.034865 12083 ts_tablet_manager.cc:1428] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
W20250113 08:57:48.035053 11931 leader_election.cc:343] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.035477 11931 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.035813 12083 tablet_bootstrap.cc:492] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:48.036249 12168 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.036223 11891 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:48.036254 12125 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.037685 12006 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:48.039021 12084 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:48.039338 12222 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:48.040170 12222 raft_consensus.cc:695] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:48.041132 12222 consensus_queue.cc:237] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.033505 12082 raft_consensus.cc:357] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.043913 12082 raft_consensus.cc:383] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.044142 12083 tablet_bootstrap.cc:654] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.044209 12082 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.045112 12082 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.039492 12084 ts_tablet_manager.cc:1397] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.005s
I20250113 08:57:48.050612 11789 catalog_manager.cc:5526] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
W20250113 08:57:48.054387 11858 consensus_peers.cc:487] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:48.060966 12082 ts_tablet_manager.cc:1428] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.041s user 0.006s sys 0.002s
I20250113 08:57:48.062042 12082 tablet_bootstrap.cc:492] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:48.067900 12123 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.074771 12084 raft_consensus.cc:357] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.076613 12084 raft_consensus.cc:383] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.082259 12084 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.083180 12084 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.093916 12083 tablet_bootstrap.cc:492] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:48.094421 12083 ts_tablet_manager.cc:1397] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.059s user 0.021s sys 0.012s
I20250113 08:57:48.095009 11789 catalog_manager.cc:5526] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:48.097353 12083 raft_consensus.cc:357] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.098057 12083 raft_consensus.cc:383] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.098305 12083 raft_consensus.cc:738] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.098923 12083 consensus_queue.cc:260] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.101241 12083 ts_tablet_manager.cc:1428] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250113 08:57:48.102149 12083 tablet_bootstrap.cc:492] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:48.091043 12082 tablet_bootstrap.cc:654] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.105631 12168 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.106266 12168 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.111099 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.111757 12082 tablet_bootstrap.cc:492] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:48.112231 12082 ts_tablet_manager.cc:1397] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.000s
I20250113 08:57:48.111991 12038 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:48.112082 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.113006 11891 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:48.113500 11931 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:48.115156 12125 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:48.115572 12125 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:48.116006 12125 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.116704 12082 raft_consensus.cc:357] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.117524 12082 raft_consensus.cc:383] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.117794 12082 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.108843 12168 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.118412 12082 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.121942 12082 ts_tablet_manager.cc:1428] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20250113 08:57:48.122001 12083 tablet_bootstrap.cc:654] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.122972 12082 tablet_bootstrap.cc:492] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:48.131099 12082 tablet_bootstrap.cc:654] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.132709 12083 tablet_bootstrap.cc:492] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:48.133194 12083 ts_tablet_manager.cc:1397] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.004s
I20250113 08:57:48.136452 12083 raft_consensus.cc:357] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.137161 12083 raft_consensus.cc:383] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.137459 12083 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.138059 12083 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.140424 12083 ts_tablet_manager.cc:1428] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250113 08:57:48.140044 12125 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.141386 12083 tablet_bootstrap.cc:492] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:48.143455 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:48.144217 11891 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.148149 12082 tablet_bootstrap.cc:492] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:48.148634 12082 ts_tablet_manager.cc:1397] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.023s
I20250113 08:57:48.150933 12083 tablet_bootstrap.cc:654] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.151820 12082 raft_consensus.cc:357] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.152554 12082 raft_consensus.cc:383] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.152810 12082 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.153537 11891 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:48.153461 12082 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.155375 11932 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:48.155887 12082 ts_tablet_manager.cc:1428] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20250113 08:57:48.156406 12168 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:48.156809 12082 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:48.157092 12168 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:48.158337 12084 ts_tablet_manager.cc:1428] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.112s user 0.002s sys 0.008s
I20250113 08:57:48.158190 12168 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.159677 12084 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:48.165023 12082 tablet_bootstrap.cc:654] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.170881 12084 tablet_bootstrap.cc:654] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.177122 12226 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.178195 12226 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.181170 12226 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:48.183264 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:48.183555 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:48.184502 12038 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.186012 12125 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.185418 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:48.188176 12006 leader_election.cc:343] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:48.189036 12007 leader_election.cc:343] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.189628 12007 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.191416 12226 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.196208 12082 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:48.196684 12082 ts_tablet_manager.cc:1397] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.015s
I20250113 08:57:48.200898 12082 raft_consensus.cc:357] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.201907 12082 raft_consensus.cc:383] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.202303 12082 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.202042 11791 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:48.203322 12082 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.206941 12084 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:48.207489 12084 ts_tablet_manager.cc:1397] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.019s
I20250113 08:57:48.210696 12082 ts_tablet_manager.cc:1428] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250113 08:57:48.210801 12084 raft_consensus.cc:357] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.211812 12082 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:48.211905 12084 raft_consensus.cc:383] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.212354 12084 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.213061 12084 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.216490 12084 ts_tablet_manager.cc:1428] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250113 08:57:48.217998 12084 tablet_bootstrap.cc:492] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:48.246403 12038 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:48.254801 12083 tablet_bootstrap.cc:492] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:48.255307 12083 ts_tablet_manager.cc:1397] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.114s user 0.031s sys 0.064s
I20250113 08:57:48.258251 12083 raft_consensus.cc:357] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.258780 12083 raft_consensus.cc:383] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.259006 12083 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.262224 12083 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.264490 12083 ts_tablet_manager.cc:1428] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s
I20250113 08:57:48.264994 12176 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.269232 12082 tablet_bootstrap.cc:654] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.270100 12083 tablet_bootstrap.cc:492] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:48.276959 12226 consensus_queue.cc:1035] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.279778 12083 tablet_bootstrap.cc:654] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.285732 12176 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.286324 12176 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.288770 12176 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.290864 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:48.295908 11858 leader_election.cc:343] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.296716 12084 tablet_bootstrap.cc:654] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:48.298060 12007 consensus_peers.cc:487] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:48.307123 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.308044 12038 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:48.312645 12084 tablet_bootstrap.cc:492] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:48.314007 11856 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:48.316258 12176 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:48.316923 12176 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:48.317592 12176 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.331825 12176 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.339648 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:48.340072 12176 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:57:48.341648 11858 leader_election.cc:343] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.345954 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:48.346953 12038 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.357605 12176 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.358275 12176 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.369174 12084 ts_tablet_manager.cc:1397] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.151s user 0.023s sys 0.023s
I20250113 08:57:48.369321 12150 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.368866 12038 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:48.372880 11856 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:48.374008 12123 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:48.374606 12123 raft_consensus.cc:695] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:48.375658 12123 consensus_queue.cc:237] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.378862 12083 tablet_bootstrap.cc:492] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:48.379365 12083 ts_tablet_manager.cc:1397] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.109s user 0.014s sys 0.009s
I20250113 08:57:48.381160 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.382081 12038 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:48.381937 12083 raft_consensus.cc:357] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.382686 12083 raft_consensus.cc:383] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.382974 12083 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.383626 11856 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:48.383587 12083 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.384718 12128 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:48.385172 12128 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:48.385962 12128 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.371578 12176 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.389940 12150 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.390538 12150 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.392741 12084 raft_consensus.cc:357] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.393502 12084 raft_consensus.cc:383] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.393784 12084 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.394882 12083 ts_tablet_manager.cc:1428] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250113 08:57:48.396739 12083 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:48.393123 12150 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:48.402769 12084 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.405068 12084 ts_tablet_manager.cc:1428] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20250113 08:57:48.408741 12084 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:48.409752 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.411862 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:48.415117 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:48.427366 12123 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
W20250113 08:57:48.430187 12006 leader_election.cc:343] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.437177 12128 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.438843 12083 tablet_bootstrap.cc:654] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.438366 12226 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250113 08:57:48.441610 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:48.442407 12042 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.444109 12128 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.445205 12082 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
W20250113 08:57:48.445022 12007 leader_election.cc:343] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.445694 12082 ts_tablet_manager.cc:1397] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.234s user 0.036s sys 0.020s
I20250113 08:57:48.445631 12007 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.447314 12226 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.444803 11790 catalog_manager.cc:5526] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
W20250113 08:57:48.451450 11858 leader_election.cc:343] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:48.453158 12007 consensus_peers.cc:487] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:48.460168 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:48.464010 12007 consensus_peers.cc:487] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:57:48.465778 11858 leader_election.cc:343] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.483355 12082 raft_consensus.cc:357] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.484671 12082 raft_consensus.cc:383] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.485095 12082 raft_consensus.cc:738] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.486256 12082 consensus_queue.cc:260] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.482344 12042 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:48.502494 12226 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.511680 12222 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.512945 12222 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.503048 11856 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:48.515573 12222 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:48.515839 12123 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:48.516928 12123 raft_consensus.cc:695] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:48.520237 12168 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.521394 12168 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.526460 12222 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.527143 12222 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.528437 12168 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.530934 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:48.531095 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.522897 12123 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:48.532896 12007 leader_election.cc:343] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:48.633062 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 100211us
I20250113 08:57:48.636616 12093 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.632612 12082 ts_tablet_manager.cc:1428] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.187s user 0.006s sys 0.002s
I20250113 08:57:48.639768 12082 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:48.640652 12222 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:48.541098 12084 tablet_bootstrap.cc:654] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.643911 12083 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:48.644376 12083 ts_tablet_manager.cc:1397] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.248s user 0.013s sys 0.017s
W20250113 08:57:48.643417 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 110244us
I20250113 08:57:48.683133 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.714793 12082 tablet_bootstrap.cc:654] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.715767 12083 raft_consensus.cc:357] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.717319 12083 raft_consensus.cc:383] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.717860 12083 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.718642 12083 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.721295 12128 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.721900 12128 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.724268 12128 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.724145 12168 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.727257 12083 ts_tablet_manager.cc:1428] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.083s user 0.015s sys 0.011s
I20250113 08:57:48.728325 12083 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:48.728964 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.729646 12038 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:48.731530 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:48.732242 11968 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:48.734124 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:48.735459 12252 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:48.735841 12252 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:48.736166 12252 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.737437 11968 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.744150 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.744072 12007 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:48.748279 12226 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:48.748769 12226 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:48.749261 12226 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:48.750013 12007 consensus_peers.cc:487] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:57:48.715041 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 79732.2us
I20250113 08:57:48.753685 12259 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.755200 12259 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.757184 11932 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.761502 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.768308 12084 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:48.771327 12084 ts_tablet_manager.cc:1397] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.364s user 0.021s sys 0.008s
I20250113 08:57:48.777530 11790 catalog_manager.cc:5526] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:48.785880 12084 raft_consensus.cc:357] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.786767 12084 raft_consensus.cc:383] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.786994 12084 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.787683 12084 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.785351 11966 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.790313 12084 ts_tablet_manager.cc:1428] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250113 08:57:48.791137 12084 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:48.798120 12168 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.798844 12259 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.801402 12083 tablet_bootstrap.cc:654] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.806106 12128 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.806738 12128 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.809762 12128 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.823639 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.824435 12226 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.826123 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:48.827579 11967 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:48.828842 12226 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:48.823084 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.830816 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:48.838778 12006 leader_election.cc:343] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.840236 12006 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.840723 12038 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:48.841293 12150 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.842198 11967 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.842342 12150 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.845904 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.846820 12082 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:48.849907 12082 ts_tablet_manager.cc:1397] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.210s user 0.025s sys 0.009s
I20250113 08:57:48.853859 12110 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:48.859625 12252 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.860071 11967 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:48.868386 12082 raft_consensus.cc:357] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.869195 12082 raft_consensus.cc:383] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.869529 12082 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.869813 12007 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:48.870167 12082 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.874814 11890 raft_consensus.cc:2388] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:48.871994 12110 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:48.904778 12084 tablet_bootstrap.cc:654] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:48.908403 12083 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:48.909073 12083 ts_tablet_manager.cc:1397] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.181s user 0.006s sys 0.021s
I20250113 08:57:48.901472 11856 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:48.911180 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:48.912355 11962 raft_consensus.cc:2388] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:48.896879 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:48.913197 11858 consensus_peers.cc:487] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:48.901398 12110 raft_consensus.cc:695] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:48.899188 11932 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.918972 12082 ts_tablet_manager.cc:1428] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.069s user 0.016s sys 0.029s
I20250113 08:57:48.918910 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:48.921289 12082 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:48.920131 12123 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:48.918892 12110 consensus_queue.cc:237] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.925786 12123 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:48.949285 12123 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:48.922649 12259 raft_consensus.cc:2743] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.920519 12176 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250113 08:57:48.945904 12093 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:48.956188 12093 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.957494 11968 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:48.939572 11858 leader_election.cc:343] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:48.895164 12252 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.915482 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:48.962622 11893 raft_consensus.cc:2388] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:57:48.964463 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:48.964871 12093 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:48.966348 12083 raft_consensus.cc:357] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.967090 12083 raft_consensus.cc:383] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.967346 12083 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:48.967957 12083 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.970876 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:48.972245 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:48.984408 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:48.973073 12269 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.985606 12038 raft_consensus.cc:2388] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:48.979910 12084 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:48.990273 12084 ts_tablet_manager.cc:1397] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.199s user 0.011s sys 0.007s
I20250113 08:57:48.989825 11931 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:48.994834 12093 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:48.998420 12084 raft_consensus.cc:357] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:48.999207 12084 raft_consensus.cc:383] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:48.999513 12084 raft_consensus.cc:738] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
W20250113 08:57:48.997911 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 110954us
I20250113 08:57:49.001147 12272 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250113 08:57:49.010746 12083 ts_tablet_manager.cc:1428] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.101s user 0.006s sys 0.001s
I20250113 08:57:49.011560 12083 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.000201 12084 consensus_queue.cc:260] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.010958 11791 catalog_manager.cc:5526] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:49.015450 12123 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.025043 12269 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:49.029914 12123 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.031643 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:49.034045 11858 leader_election.cc:343] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.044302 12084 ts_tablet_manager.cc:1428] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.054s user 0.005s sys 0.001s
I20250113 08:57:49.057834 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:49.058879 12041 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.059669 12084 tablet_bootstrap.cc:492] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:49.094442 12083 tablet_bootstrap.cc:654] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.096037 12082 tablet_bootstrap.cc:654] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.096067 12041 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:49.109519 11856 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:49.112126 12252 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:49.112869 12252 raft_consensus.cc:695] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:49.114243 12252 consensus_queue.cc:237] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.116304 12226 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.117863 12226 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.120240 12226 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.123214 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:49.124050 11890 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:49.124558 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.125413 11966 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:49.125989 12006 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:49.127271 12270 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.127705 12270 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.128116 12270 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.232048 12270 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.238000 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:49.238335 12270 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.239892 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:49.240357 12270 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.240602 11963 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.238977 11890 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.240957 12270 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.249033 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:49.249223 12270 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.250078 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:49.250756 12006 leader_election.cc:343] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:49.251540 12007 leader_election.cc:343] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.252207 12007 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:49.251777 12084 tablet_bootstrap.cc:654] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.254525 12082 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:49.254985 12082 ts_tablet_manager.cc:1397] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.334s user 0.021s sys 0.020s
I20250113 08:57:49.258695 12283 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:49.259825 12083 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.260082 12084 tablet_bootstrap.cc:492] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:49.260274 12083 ts_tablet_manager.cc:1397] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.249s user 0.039s sys 0.037s
I20250113 08:57:49.260474 12084 ts_tablet_manager.cc:1397] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.201s user 0.005s sys 0.010s
I20250113 08:57:49.262794 12083 raft_consensus.cc:357] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.263526 12083 raft_consensus.cc:383] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.263780 12083 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.265636 12084 raft_consensus.cc:357] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.266685 12084 raft_consensus.cc:383] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.264494 12083 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.267014 12084 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.266877 12082 raft_consensus.cc:357] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.266455 11890 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:49.270356 12082 raft_consensus.cc:383] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.271103 12082 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.271670 12006 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:49.272583 12083 ts_tablet_manager.cc:1428] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.012s user 0.002s sys 0.006s
I20250113 08:57:49.272127 12082 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.272835 12270 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:49.273684 12270 raft_consensus.cc:695] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:49.270200 12084 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.274920 12082 ts_tablet_manager.cc:1428] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.020s user 0.005s sys 0.005s
I20250113 08:57:49.275992 12082 tablet_bootstrap.cc:492] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:49.269524 11789 catalog_manager.cc:5526] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:49.279454 12083 tablet_bootstrap.cc:492] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.282894 12084 ts_tablet_manager.cc:1428] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250113 08:57:49.283924 12084 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:49.288584 12083 tablet_bootstrap.cc:654] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.294122 11963 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:49.290639 12270 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.307557 12084 tablet_bootstrap.cc:654] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.308562 11963 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.315150 12082 tablet_bootstrap.cc:654] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.318230 11791 catalog_manager.cc:5526] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:49.319664 12083 tablet_bootstrap.cc:492] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.320413 12083 ts_tablet_manager.cc:1397] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.008s
I20250113 08:57:49.328565 12270 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.329130 12270 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.332280 12083 raft_consensus.cc:357] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.332888 12270 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.333485 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.333072 12083 raft_consensus.cc:383] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.334177 12083 raft_consensus.cc:738] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
W20250113 08:57:49.335217 12007 leader_election.cc:343] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.335318 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:49.337213 12006 leader_election.cc:343] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.337910 12006 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:49.339705 12270 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:49.335275 12083 consensus_queue.cc:260] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.350726 12082 tablet_bootstrap.cc:492] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:49.351223 12082 ts_tablet_manager.cc:1397] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.010s
I20250113 08:57:49.354497 12083 ts_tablet_manager.cc:1428] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.034s user 0.006s sys 0.002s
I20250113 08:57:49.355466 12083 tablet_bootstrap.cc:492] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.358541 12082 raft_consensus.cc:357] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.359351 12082 raft_consensus.cc:383] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.359639 12082 raft_consensus.cc:738] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.360278 12082 consensus_queue.cc:260] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.362632 12082 ts_tablet_manager.cc:1428] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250113 08:57:49.369151 12084 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:49.369577 12082 tablet_bootstrap.cc:492] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:49.369660 12084 ts_tablet_manager.cc:1397] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.086s user 0.014s sys 0.030s
I20250113 08:57:49.373212 12084 raft_consensus.cc:357] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.374460 12084 raft_consensus.cc:383] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.374761 12084 raft_consensus.cc:738] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.375440 12084 consensus_queue.cc:260] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.378501 12084 ts_tablet_manager.cc:1428] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20250113 08:57:49.379346 12084 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:49.388329 12084 tablet_bootstrap.cc:654] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.389343 12082 tablet_bootstrap.cc:654] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.391685 12083 tablet_bootstrap.cc:654] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.402747 12084 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:49.403285 12084 ts_tablet_manager.cc:1397] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.016s
I20250113 08:57:49.404003 12082 tablet_bootstrap.cc:492] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:49.404479 12083 tablet_bootstrap.cc:492] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.404505 12082 ts_tablet_manager.cc:1397] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.014s
I20250113 08:57:49.404934 12083 ts_tablet_manager.cc:1397] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.017s
I20250113 08:57:49.407774 12084 raft_consensus.cc:357] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.408584 12084 raft_consensus.cc:383] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.408854 12084 raft_consensus.cc:738] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.408879 12082 raft_consensus.cc:357] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.409729 12082 raft_consensus.cc:383] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.409631 12083 raft_consensus.cc:357] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.410122 12082 raft_consensus.cc:738] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.409749 12084 consensus_queue.cc:260] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.410331 12083 raft_consensus.cc:383] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.410624 12083 raft_consensus.cc:738] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.410764 12082 consensus_queue.cc:260] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.411266 12083 consensus_queue.cc:260] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.415195 12270 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:49.417961 12083 ts_tablet_manager.cc:1428] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
I20250113 08:57:49.418802 12084 ts_tablet_manager.cc:1428] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
I20250113 08:57:49.419088 12083 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.419667 12084 tablet_bootstrap.cc:492] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:49.413435 12082 ts_tablet_manager.cc:1428] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20250113 08:57:49.428544 12082 tablet_bootstrap.cc:492] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:49.431988 12084 tablet_bootstrap.cc:654] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.437417 12082 tablet_bootstrap.cc:654] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.429036 12083 tablet_bootstrap.cc:654] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.457757 11968 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.464210 11966 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.481415 12226 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:49.496661 12083 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.497066 12083 ts_tablet_manager.cc:1397] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.012s
I20250113 08:57:49.499287 12083 raft_consensus.cc:357] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.499776 12083 raft_consensus.cc:383] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.500053 12083 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.500708 12083 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.511016 12083 ts_tablet_manager.cc:1428] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20250113 08:57:49.512244 12083 tablet_bootstrap.cc:492] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.544742 12083 tablet_bootstrap.cc:654] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.566018 12083 tablet_bootstrap.cc:492] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.566493 12083 ts_tablet_manager.cc:1397] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.008s
I20250113 08:57:49.574355 12082 tablet_bootstrap.cc:492] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:49.574862 12082 ts_tablet_manager.cc:1397] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.147s user 0.042s sys 0.100s
I20250113 08:57:49.569562 12083 raft_consensus.cc:357] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.576846 12083 raft_consensus.cc:383] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.577175 12083 raft_consensus.cc:738] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.577899 12083 consensus_queue.cc:260] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.579046 12082 raft_consensus.cc:357] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.579807 12082 raft_consensus.cc:383] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.580091 12082 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.580711 12082 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.582690 12082 ts_tablet_manager.cc:1428] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20250113 08:57:49.583525 12082 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:49.589556 12083 ts_tablet_manager.cc:1428] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20250113 08:57:49.590550 12083 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.591070 12082 tablet_bootstrap.cc:654] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.595417 12084 tablet_bootstrap.cc:492] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:49.597787 12084 ts_tablet_manager.cc:1397] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.178s user 0.010s sys 0.014s
I20250113 08:57:49.600931 12083 tablet_bootstrap.cc:654] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.600833 12084 raft_consensus.cc:357] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.604496 12084 raft_consensus.cc:383] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.619227 12084 raft_consensus.cc:738] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.611581 12082 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:49.620791 12082 ts_tablet_manager.cc:1397] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.011s
I20250113 08:57:49.620283 12084 consensus_queue.cc:260] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.623255 12084 ts_tablet_manager.cc:1428] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.025s user 0.001s sys 0.005s
I20250113 08:57:49.624007 12082 raft_consensus.cc:357] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.624722 12082 raft_consensus.cc:383] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.624956 12084 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:49.624969 12082 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.626322 12082 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.630270 12082 ts_tablet_manager.cc:1428] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20250113 08:57:49.631230 12082 tablet_bootstrap.cc:492] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:49.633579 12084 tablet_bootstrap.cc:654] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.639008 12082 tablet_bootstrap.cc:654] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.654551 12083 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.655189 12083 ts_tablet_manager.cc:1397] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.065s user 0.026s sys 0.019s
I20250113 08:57:49.655992 12252 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:49.658550 12083 raft_consensus.cc:357] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.659471 12083 raft_consensus.cc:383] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.659776 12083 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.660545 12083 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.662501 12083 ts_tablet_manager.cc:1428] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250113 08:57:49.663422 12083 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.664252 12084 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:49.664351 11963 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.664728 12084 ts_tablet_manager.cc:1397] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.001s
I20250113 08:57:49.667883 12084 raft_consensus.cc:357] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.668587 12084 raft_consensus.cc:383] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.668578 12252 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.668934 12084 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.669456 12252 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.674157 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.674759 12252 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:57:49.676121 11858 leader_election.cc:343] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.678663 12082 tablet_bootstrap.cc:492] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:49.679147 12082 ts_tablet_manager.cc:1397] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.048s user 0.018s sys 0.005s
I20250113 08:57:49.679319 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:49.679983 12038 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:49.682237 12082 raft_consensus.cc:357] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.682994 12082 raft_consensus.cc:383] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.683290 12082 raft_consensus.cc:738] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.684073 12082 consensus_queue.cc:260] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.684338 11856 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:49.669641 12084 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.671082 12083 tablet_bootstrap.cc:654] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.685576 12123 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.685998 12123 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.686578 12082 ts_tablet_manager.cc:1428] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250113 08:57:49.687608 12082 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:49.686450 12123 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.689850 12084 ts_tablet_manager.cc:1428] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.025s user 0.004s sys 0.005s
I20250113 08:57:49.691977 12084 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:49.702628 12252 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:49.706470 12226 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.707499 12226 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.711171 12226 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.712165 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:49.705430 12123 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:49.713974 12006 leader_election.cc:343] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.720548 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.721798 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:49.722548 12038 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:49.723919 12007 leader_election.cc:343] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.724468 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:49.724640 12007 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:49.722416 12252 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.730027 12226 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:49.730532 11858 leader_election.cc:343] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.726596 12123 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.737279 12252 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.738934 12252 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.740165 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:49.740303 12270 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:49.740451 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.740916 12042 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:49.741598 12270 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.741453 11967 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:49.742117 12270 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.742905 11856 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:49.743844 12252 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.744222 12270 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.744277 12252 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.744879 12252 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.745612 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:49.746412 11893 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:49.736874 12038 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:49.748447 12006 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:49.748986 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.750094 11856 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:49.751506 12123 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:49.752053 12123 raft_consensus.cc:695] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:57:49.750835 12007 leader_election.cc:343] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.753115 12123 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.751945 12270 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.756319 12270 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.756819 12270 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.759924 12083 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.760672 12083 ts_tablet_manager.cc:1397] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.097s user 0.017s sys 0.020s
I20250113 08:57:49.761847 12084 tablet_bootstrap.cc:654] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.761772 12252 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.795279 12082 tablet_bootstrap.cc:654] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:49.797333 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45171.9us
I20250113 08:57:49.800397 12252 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.807301 12083 raft_consensus.cc:357] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.808967 12083 raft_consensus.cc:383] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.809351 12083 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.805044 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:49.810487 11967 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.803128 11789 catalog_manager.cc:5526] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:49.815716 12083 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.822568 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:49.823459 12042 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.855283 12083 ts_tablet_manager.cc:1428] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.094s user 0.009s sys 0.000s
I20250113 08:57:49.856554 12083 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:49.860476 11968 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.862365 12310 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.863466 12310 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.865537 12310 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.858891 12270 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.872638 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.872973 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:49.875949 12039 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:49.877092 12150 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.877779 12150 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.879953 12150 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.880438 12150 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.880662 12270 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.885735 12310 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.886256 12310 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.880854 12150 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.877158 11856 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:49.901495 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:49.907889 11893 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:49.908377 12150 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:49.908676 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:49.911901 12084 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:49.912310 12084 ts_tablet_manager.cc:1397] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.221s user 0.058s sys 0.043s
I20250113 08:57:49.888036 11967 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:49.915089 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:49.891635 12310 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.917122 11890 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.909947 12128 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.920921 12083 tablet_bootstrap.cc:654] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:49.926137 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.920725 12128 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.927332 12128 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.919613 12123 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:49.931308 12123 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.932117 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:49.932819 12039 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:49.934412 12123 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.935825 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.937134 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.938479 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:49.921339 11858 leader_election.cc:343] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.940848 12006 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:49.944067 12006 leader_election.cc:343] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.948832 12082 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:49.949378 12082 ts_tablet_manager.cc:1397] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.262s user 0.020s sys 0.007s
I20250113 08:57:49.949856 12083 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:49.950345 12083 ts_tablet_manager.cc:1397] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.094s user 0.010s sys 0.003s
I20250113 08:57:49.956079 12082 raft_consensus.cc:357] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.956842 12082 raft_consensus.cc:383] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.957091 12082 raft_consensus.cc:738] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.957700 12082 consensus_queue.cc:260] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.953238 12128 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.959097 11856 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:49.959906 12128 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:49.960140 12123 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.960879 12082 ts_tablet_manager.cc:1428] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20250113 08:57:49.962247 12082 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:49.951268 12150 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.960552 12123 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.971343 12083 raft_consensus.cc:357] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.971863 12083 raft_consensus.cc:383] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.972085 12083 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.968461 12123 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.974408 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:49.976317 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:49.976963 12040 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.977087 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:49.966755 12150 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.977798 12038 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:49.978155 12150 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:49.980758 11890 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:49.982672 12083 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.984532 12083 ts_tablet_manager.cc:1428] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
W20250113 08:57:49.985492 12007 leader_election.cc:343] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.982501 11856 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:49.986477 12007 leader_election.cc:343] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.986876 12007 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:49.987151 12084 raft_consensus.cc:357] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:49.987746 12084 raft_consensus.cc:383] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:49.987645 12322 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:49.987980 12007 leader_election.cc:343] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.988088 12084 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:49.989737 12128 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:49.994972 12128 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:49.995410 12128 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:49.997331 11858 leader_election.cc:343] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:49.998580 11858 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:49.991272 12083 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:50.000219 12084 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.012042 12252 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:50.012879 12252 raft_consensus.cc:695] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:50.014223 12252 consensus_queue.cc:237] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.018954 12040 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:50.021442 12123 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.049232 12084 ts_tablet_manager.cc:1428] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.137s user 0.007s sys 0.002s
W20250113 08:57:50.051308 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 61178.9us
I20250113 08:57:50.060160 12084 tablet_bootstrap.cc:492] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
W20250113 08:57:50.050486 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 52030.6us
I20250113 08:57:50.067719 12082 tablet_bootstrap.cc:654] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.077950 12042 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:50.088348 12123 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.090436 12128 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.102973 12128 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.103649 12259 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.104317 12259 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.104985 11856 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:50.106896 12259 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.110061 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.111001 12270 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.111485 12270 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:50.081156 11858 leader_election.cc:343] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.112556 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:50.114212 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:50.110117 12128 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:50.115165 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.115935 12042 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:50.116730 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:50.115468 12128 raft_consensus.cc:695] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:50.121806 11931 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.122090 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:50.123732 12093 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:50.104000 12150 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.113852 12270 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:50.113857 12039 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:50.121577 12128 consensus_queue.cc:237] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.142643 12150 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:50.139915 12006 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:50.138072 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:50.149927 12040 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:50.154729 12083 tablet_bootstrap.cc:654] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.149542 12150 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:50.163262 12150 raft_consensus.cc:695] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:50.166463 12093 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.166352 12150 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.167420 12084 tablet_bootstrap.cc:654] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.170363 12082 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:50.171177 12082 ts_tablet_manager.cc:1397] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.209s user 0.015s sys 0.004s
I20250113 08:57:50.170627 11790 catalog_manager.cc:5526] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:50.179874 12082 raft_consensus.cc:357] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.180940 12082 raft_consensus.cc:383] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.182745 12082 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.183164 12039 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:50.183490 12082 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.184300 11856 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:50.186419 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.194123 12084 tablet_bootstrap.cc:492] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:50.194614 12084 ts_tablet_manager.cc:1397] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.135s user 0.005s sys 0.011s
I20250113 08:57:50.197034 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:50.198194 11891 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:50.203076 12270 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:50.205624 12082 ts_tablet_manager.cc:1428] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.034s user 0.005s sys 0.002s
W20250113 08:57:50.206189 11858 leader_election.cc:343] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.205489 11789 catalog_manager.cc:5526] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:50.206576 12082 tablet_bootstrap.cc:492] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:50.187338 12128 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:50.210678 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:50.213372 12128 raft_consensus.cc:695] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:50.215293 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:50.223803 11858 leader_election.cc:343] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:50.226765 11858 leader_election.cc:343] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.224483 12128 consensus_queue.cc:237] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.227419 11858 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:50.229925 12252 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.223446 12084 raft_consensus.cc:357] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.232481 12084 raft_consensus.cc:383] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.232854 12084 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.234221 12084 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.233282 11790 catalog_manager.cc:5526] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
W20250113 08:57:50.237612 12006 leader_election.cc:343] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.240808 12330 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.242377 12330 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.244848 12123 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:50.250344 12330 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.252100 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.253691 12084 ts_tablet_manager.cc:1428] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.059s user 0.002s sys 0.007s
I20250113 08:57:50.254483 12084 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
W20250113 08:57:50.262060 11856 leader_election.cc:343] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.262871 12083 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:50.263227 12083 ts_tablet_manager.cc:1397] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.272s user 0.014s sys 0.004s
W20250113 08:57:50.263875 12007 leader_election.cc:343] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.268914 12007 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.270169 12318 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:50.271291 12007 leader_election.cc:343] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.271795 12007 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.272675 12329 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.272655 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:50.287648 12083 raft_consensus.cc:357] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.288476 12083 raft_consensus.cc:383] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.288760 12083 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.289436 12083 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.297098 12083 ts_tablet_manager.cc:1428] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.034s user 0.007s sys 0.002s
I20250113 08:57:50.298017 12083 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:50.304967 12270 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.304883 11789 catalog_manager.cc:5526] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:50.305599 12270 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.307962 12270 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:57:50.310309 11858 leader_election.cc:343] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.311832 12082 tablet_bootstrap.cc:654] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:50.313669 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:50.317605 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:50.321519 12007 leader_election.cc:343] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.325435 11858 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:50.326395 12330 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.330889 12084 tablet_bootstrap.cc:654] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.336264 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:50.342556 12006 leader_election.cc:343] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.343269 12006 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.344446 12270 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.361438 12123 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:50.371204 12084 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:50.371693 12082 tablet_bootstrap.cc:492] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:50.371738 12084 ts_tablet_manager.cc:1397] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.117s user 0.011s sys 0.009s
I20250113 08:57:50.372242 12082 ts_tablet_manager.cc:1397] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.166s user 0.009s sys 0.014s
I20250113 08:57:50.374938 12084 raft_consensus.cc:357] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.375514 12084 raft_consensus.cc:383] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.375605 12083 tablet_bootstrap.cc:654] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.375732 12084 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.376459 12084 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.378872 12084 ts_tablet_manager.cc:1428] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:57:50.379776 12084 tablet_bootstrap.cc:492] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:50.387122 12259 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.387460 12084 tablet_bootstrap.cc:654] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.387974 12259 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.398506 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.399551 12039 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:50.399576 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.414211 12082 raft_consensus.cc:357] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.415161 12082 raft_consensus.cc:383] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.415498 12082 raft_consensus.cc:738] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.417416 12259 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.417420 12082 consensus_queue.cc:260] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.420832 12082 ts_tablet_manager.cc:1428] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.048s user 0.009s sys 0.000s
I20250113 08:57:50.433385 12082 tablet_bootstrap.cc:492] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:50.437664 12128 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.439519 11932 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.444900 12084 tablet_bootstrap.cc:492] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:50.446823 12084 ts_tablet_manager.cc:1397] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.067s user 0.017s sys 0.023s
I20250113 08:57:50.446727 12093 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:50.451014 12084 raft_consensus.cc:357] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.452250 12084 raft_consensus.cc:383] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.452576 12084 raft_consensus.cc:738] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.453359 12084 consensus_queue.cc:260] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.454434 12259 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.455799 12084 ts_tablet_manager.cc:1428] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250113 08:57:50.445472 12128 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.456815 12084 tablet_bootstrap.cc:492] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:50.460124 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.461237 12039 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:50.464542 12084 tablet_bootstrap.cc:654] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.467974 11856 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:50.469781 12128 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.469947 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:50.470523 12128 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:50.471030 12128 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:50.471460 12128 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:50.475708 11858 leader_election.cc:343] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.455178 12259 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.480257 12084 tablet_bootstrap.cc:492] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:50.481906 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.482332 12259 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.484669 12084 ts_tablet_manager.cc:1397] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.009s
I20250113 08:57:50.536716 12345 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.537667 12345 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.553701 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.557827 12084 raft_consensus.cc:357] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.563333 12084 raft_consensus.cc:383] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.560407 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.567020 11893 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:50.567281 12082 tablet_bootstrap.cc:654] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.561652 11890 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:50.559600 12345 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.563736 12084 raft_consensus.cc:738] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.571507 12084 consensus_queue.cc:260] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.573623 12084 ts_tablet_manager.cc:1428] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.088s user 0.012s sys 0.008s
I20250113 08:57:50.575742 12084 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:50.581127 11967 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:50.587628 12093 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.595407 11968 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:50.619164 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.616214 12330 consensus_queue.cc:1035] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250113 08:57:50.625500 12040 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:50.629518 12083 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:50.629910 12083 ts_tablet_manager.cc:1397] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.332s user 0.090s sys 0.111s
I20250113 08:57:50.632010 12252 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:50.635330 12150 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:50.639995 12093 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.636996 12083 raft_consensus.cc:357] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.640764 12083 raft_consensus.cc:383] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.641178 12083 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.640578 12093 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.641956 12083 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.646811 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.643836 12093 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.628696 11931 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.653585 12322 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.654700 12322 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.657703 12128 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.660579 12083 ts_tablet_manager.cc:1428] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20250113 08:57:50.661942 12083 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:50.665926 12128 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.668298 12082 tablet_bootstrap.cc:492] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:50.668776 12082 ts_tablet_manager.cc:1397] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.236s user 0.022s sys 0.006s
I20250113 08:57:50.679785 12093 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.679095 11931 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.681911 12347 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:50.703531 12322 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:50.680847 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:50.710640 12039 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:50.718678 12084 tablet_bootstrap.cc:654] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.719698 12347 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.722584 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:50.728102 12039 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:50.731622 11856 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:50.735028 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.722602 12226 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.738193 12226 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.740681 12226 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:57:50.741730 12006 leader_election.cc:343] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.744274 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.744872 12330 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:50.745545 12330 raft_consensus.cc:695] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:50.745191 12040 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:50.729682 12128 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250113 08:57:50.746582 12330 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.749835 11931 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.744313 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.754832 12084 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:50.757004 12084 ts_tablet_manager.cc:1397] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.181s user 0.012s sys 0.004s
I20250113 08:57:50.756928 12082 raft_consensus.cc:357] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.757826 12082 raft_consensus.cc:383] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.758101 12082 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.758674 12082 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:50.761132 11858 leader_election.cc:343] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:50.763386 12006 leader_election.cc:343] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.765043 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:50.766865 12082 ts_tablet_manager.cc:1428] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.098s user 0.004s sys 0.004s
I20250113 08:57:50.770179 12083 tablet_bootstrap.cc:654] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.767453 12350 consensus_queue.cc:1035] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:50.771312 12082 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:50.773545 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:50.768692 12347 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:57:50.776715 12347 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:50.782445 12347 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:50.779270 12226 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250113 08:57:50.784785 12007 leader_election.cc:343] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.787014 12007 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.788599 12150 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:50.789072 12007 leader_election.cc:343] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:50.789806 12007 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:50.790954 12123 consensus_queue.cc:1035] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:50.801678 12150 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:50.808714 12084 raft_consensus.cc:357] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.809988 12084 raft_consensus.cc:383] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.810406 12084 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.811342 12084 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.838166 12347 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.835227 12084 ts_tablet_manager.cc:1428] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.078s user 0.008s sys 0.000s
I20250113 08:57:50.842133 11791 catalog_manager.cc:5526] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:50.847043 12084 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:50.856125 12128 consensus_queue.cc:1035] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250113 08:57:50.884744 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 93602.2us
I20250113 08:57:50.902491 12082 tablet_bootstrap.cc:654] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.908401 12330 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.919232 12330 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.955607 12330 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.946566 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:50.978928 12083 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:50.978816 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.979485 12083 ts_tablet_manager.cc:1397] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.318s user 0.033s sys 0.049s
I20250113 08:57:50.968724 12345 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:50.979693 12042 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:50.980098 12345 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.984659 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:50.984742 12345 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:50.980672 12084 tablet_bootstrap.cc:654] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:50.991875 12083 raft_consensus.cc:357] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:50.992434 12083 raft_consensus.cc:383] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:50.992959 12083 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:50.995611 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:50.998219 12038 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:50.996300 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:50.996798 11856 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:50.994920 12083 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:50.997032 11932 leader_election.cc:343] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.002437 12350 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.002827 12083 ts_tablet_manager.cc:1428] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.023s user 0.008s sys 0.002s
I20250113 08:57:51.003031 12350 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.003612 12350 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.004817 11966 raft_consensus.cc:1235] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:57:50.994204 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.010334 11931 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.003799 12083 tablet_bootstrap.cc:492] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:50.990984 12347 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.013340 12093 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.013828 12093 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.014235 12093 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.015666 11931 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:51.020395 11858 leader_election.cc:343] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.029505 12123 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.030099 12123 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.038368 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.038694 12326 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:51.040019 11856 leader_election.cc:343] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.017702 12345 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:57:51.039321 12326 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.041441 12123 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.048354 12252 consensus_queue.cc:1046] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:51.050382 12252 raft_consensus.cc:3049] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:51.050835 12252 raft_consensus.cc:738] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:51.050419 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:51.053649 11858 leader_election.cc:343] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.054122 11858 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:51.056761 12128 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:51.051841 12252 consensus_queue.cc:260] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.058110 12252 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:51.063694 12083 tablet_bootstrap.cc:654] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.068214 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:51.069906 12007 leader_election.cc:343] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.067351 12093 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.075867 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:51.077994 12093 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.078650 12082 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:51.079200 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:51.079734 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.079202 12082 ts_tablet_manager.cc:1397] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.308s user 0.016s sys 0.011s
I20250113 08:57:51.080472 12040 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:51.080834 11932 leader_election.cc:343] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:51.083452 12006 leader_election.cc:343] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.084007 12006 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.085793 12084 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:51.086545 12084 ts_tablet_manager.cc:1397] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.240s user 0.012s sys 0.005s
I20250113 08:57:51.085702 12350 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.086783 12226 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:51.090639 12350 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.087942 12083 tablet_bootstrap.cc:492] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:51.090945 12040 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:51.091421 12083 ts_tablet_manager.cc:1397] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.088s user 0.012s sys 0.006s
I20250113 08:57:51.091328 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.092016 12038 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.093466 12084 raft_consensus.cc:357] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.094358 12083 raft_consensus.cc:357] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.100085 12084 raft_consensus.cc:383] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.100283 12083 raft_consensus.cc:383] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.099531 12082 raft_consensus.cc:357] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.100385 12084 raft_consensus.cc:738] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.100801 12083 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.101297 12084 consensus_queue.cc:260] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.101572 12083 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.098573 11931 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.103699 12084 ts_tablet_manager.cc:1428] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250113 08:57:51.103839 12083 ts_tablet_manager.cc:1428] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250113 08:57:51.105082 12083 tablet_bootstrap.cc:492] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:51.103812 12347 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.100806 12082 raft_consensus.cc:383] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.109304 12347 raft_consensus.cc:695] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:51.082347 12326 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.110425 12347 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.109308 12082 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.113508 12083 tablet_bootstrap.cc:654] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.114756 12082 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.116596 12082 ts_tablet_manager.cc:1428] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20250113 08:57:51.118737 12082 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:51.120268 12083 tablet_bootstrap.cc:492] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:51.120760 12083 ts_tablet_manager.cc:1397] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20250113 08:57:51.111018 12038 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:51.124863 12083 raft_consensus.cc:357] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.125216 11856 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.125610 12083 raft_consensus.cc:383] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.125936 12083 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.126420 12252 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.127069 12252 raft_consensus.cc:695] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:51.129390 12083 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.128160 12252 consensus_queue.cc:237] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.132543 12083 ts_tablet_manager.cc:1428] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20250113 08:57:51.133476 12083 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:51.134553 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:51.137625 12084 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:51.141173 12082 tablet_bootstrap.cc:654] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.142416 12083 tablet_bootstrap.cc:654] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:51.146064 11858 leader_election.cc:343] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.155761 12082 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:51.156278 12082 ts_tablet_manager.cc:1397] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.011s
I20250113 08:57:51.166368 12084 tablet_bootstrap.cc:654] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.182041 12083 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:51.182494 12083 ts_tablet_manager.cc:1397] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.009s
I20250113 08:57:51.182610 12082 raft_consensus.cc:357] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.183398 12082 raft_consensus.cc:383] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.183638 12082 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.184195 12082 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.184602 12083 raft_consensus.cc:357] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.185294 12083 raft_consensus.cc:383] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.185585 12083 raft_consensus.cc:738] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.186131 12083 consensus_queue.cc:260] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.186435 12082 ts_tablet_manager.cc:1428] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20250113 08:57:51.187386 12082 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:51.187831 12083 ts_tablet_manager.cc:1428] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250113 08:57:51.188551 12083 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:51.189134 11791 catalog_manager.cc:5526] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:51.200255 12082 tablet_bootstrap.cc:654] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.203785 12084 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:51.204150 12350 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.204313 12084 ts_tablet_manager.cc:1397] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.002s
I20250113 08:57:51.204632 12350 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.205698 12083 tablet_bootstrap.cc:654] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.206881 12084 raft_consensus.cc:357] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.207599 12084 raft_consensus.cc:383] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.207845 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.207860 12084 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.208590 12038 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:57:51.208773 12084 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.210481 12350 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.211611 12084 ts_tablet_manager.cc:1428] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250113 08:57:51.211149 11789 catalog_manager.cc:5526] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:51.213402 12084 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:51.216001 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.225181 11858 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:51.226325 12350 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.239284 12084 tablet_bootstrap.cc:654] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.235246 12350 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:51.245024 12082 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:51.245561 12082 ts_tablet_manager.cc:1397] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.007s
I20250113 08:57:51.249761 12082 raft_consensus.cc:357] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.250502 12082 raft_consensus.cc:383] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.250774 12082 raft_consensus.cc:738] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.251534 12082 consensus_queue.cc:260] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.254026 12082 ts_tablet_manager.cc:1428] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250113 08:57:51.255313 12082 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:51.272670 12082 tablet_bootstrap.cc:654] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:51.275264 11858 consensus_peers.cc:487] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:51.286944 12084 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:51.287470 12084 ts_tablet_manager.cc:1397] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.074s user 0.018s sys 0.023s
I20250113 08:57:51.288095 12083 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:51.288091 12082 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:51.288712 12083 ts_tablet_manager.cc:1397] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.100s user 0.023s sys 0.032s
I20250113 08:57:51.288827 12082 ts_tablet_manager.cc:1397] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.009s
I20250113 08:57:51.290589 12084 raft_consensus.cc:357] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.291064 12082 raft_consensus.cc:357] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.291314 12084 raft_consensus.cc:383] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.291621 12082 raft_consensus.cc:383] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.291663 12084 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.292021 12082 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.291790 12083 raft_consensus.cc:357] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.292543 12083 raft_consensus.cc:383] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.292810 12083 raft_consensus.cc:738] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.292913 12084 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.293531 12083 consensus_queue.cc:260] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.294165 12082 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.296782 12084 ts_tablet_manager.cc:1428] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250113 08:57:51.296810 12083 ts_tablet_manager.cc:1428] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250113 08:57:51.297889 12083 tablet_bootstrap.cc:492] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:51.298293 12082 ts_tablet_manager.cc:1428] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20250113 08:57:51.299026 12082 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:51.305192 12083 tablet_bootstrap.cc:654] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.306306 12082 tablet_bootstrap.cc:654] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.307812 12226 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.308388 12226 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.311782 12084 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:51.314281 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:51.319175 12006 leader_election.cc:343] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.320292 12150 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.320880 12150 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.323403 12350 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:51.335464 12150 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.337625 12226 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.338439 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.339138 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.339660 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:51.340791 11890 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:51.342466 12006 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:51.343214 12007 leader_election.cc:343] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.343794 12084 tablet_bootstrap.cc:654] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.344251 12150 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250113 08:57:51.344324 12007 leader_election.cc:343] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.345533 12150 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.346447 12150 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.346153 12007 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.348698 12082 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:51.347172 12226 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.349691 12326 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:51.349162 12082 ts_tablet_manager.cc:1397] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.009s
I20250113 08:57:51.349535 12226 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.358526 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:51.359396 11890 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:51.361830 12006 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.365157 12326 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.360097 12226 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.361900 12083 tablet_bootstrap.cc:492] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:51.365742 12326 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.366266 12083 ts_tablet_manager.cc:1397] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.010s
I20250113 08:57:51.366194 12326 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.369522 12083 raft_consensus.cc:357] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.369230 12150 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.370723 12322 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.370334 12083 raft_consensus.cc:383] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.371259 12322 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.372116 12083 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.372750 12083 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.373778 12322 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.360492 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.375200 12083 ts_tablet_manager.cc:1428] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250113 08:57:51.375528 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:51.376142 11893 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:51.376336 12083 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
W20250113 08:57:51.376827 12007 leader_election.cc:343] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.377203 12006 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.378551 12322 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.378947 12322 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.378741 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.379283 12322 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.379549 11966 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:51.373791 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:51.382263 11890 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.386961 12326 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.363394 12082 raft_consensus.cc:357] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.404114 12082 raft_consensus.cc:383] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.404495 12082 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.405063 12082 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.419144 12150 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:57:51.445307 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:51.432099 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:51.433347 12322 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.426301 11890 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:51.456789 12083 tablet_bootstrap.cc:654] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:51.456581 12007 leader_election.cc:343] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.426007 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:51.457918 12322 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:57:51.458673 12007 leader_election.cc:343] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.459797 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:51.427596 12326 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.460364 11966 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.435765 12082 ts_tablet_manager.cc:1428] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.085s user 0.009s sys 0.000s
I20250113 08:57:51.462013 12082 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:51.466670 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:51.467343 11893 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.467783 12084 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:51.468250 12084 ts_tablet_manager.cc:1397] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.157s user 0.011s sys 0.019s
I20250113 08:57:51.469000 11966 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:51.470366 12083 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:51.470371 12007 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:51.471319 12083 ts_tablet_manager.cc:1397] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.007s
I20250113 08:57:51.471269 12084 raft_consensus.cc:357] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.471808 12326 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.472359 12347 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.473197 12326 raft_consensus.cc:695] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:51.473238 12347 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.474551 11893 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:51.474398 12326 consensus_queue.cc:237] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.474591 12083 raft_consensus.cc:357] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.475486 12083 raft_consensus.cc:383] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.475813 12083 raft_consensus.cc:738] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.475823 12347 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.476796 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:51.476676 12083 consensus_queue.cc:260] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.472911 12084 raft_consensus.cc:383] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250113 08:57:51.480175 11932 leader_election.cc:343] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.480177 12084 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.481406 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:51.482906 11890 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.466733 12006 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:51.485854 12322 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.481884 12083 ts_tablet_manager.cc:1428] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20250113 08:57:51.483953 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.486444 12322 raft_consensus.cc:695] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:51.486644 12038 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:51.487532 12322 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.488970 12082 tablet_bootstrap.cc:654] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.488533 12083 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:51.493270 11931 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.495018 12347 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.495554 12347 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.496137 12347 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.488500 12084 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.501138 12084 ts_tablet_manager.cc:1428] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.033s user 0.006s sys 0.002s
I20250113 08:57:51.502343 12083 tablet_bootstrap.cc:654] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.502195 12345 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.502879 12345 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.505380 12345 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.508595 12084 tablet_bootstrap.cc:492] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:51.509145 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:51.510588 11932 leader_election.cc:343] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.512916 12350 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.513613 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.513526 12350 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.514564 12039 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:51.516700 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.517549 11962 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:51.517602 11931 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.518815 11858 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.519884 12123 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.520020 12350 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.520236 12123 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.521301 12123 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.521119 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.522035 12039 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:51.515245 12347 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.534169 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:51.529410 12345 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.537729 12345 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:57:51.537837 11932 leader_election.cc:343] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.538192 12345 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.538723 12123 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.533385 12082 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:51.545512 12123 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.546361 12082 ts_tablet_manager.cc:1397] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.011s
I20250113 08:57:51.549623 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.550390 12038 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.540010 12347 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.545123 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:51.553275 12330 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.553681 11962 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.553881 12330 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.562425 12330 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.564348 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.564891 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.565218 12042 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:51.568352 11856 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:51.568830 11858 leader_election.cc:343] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.569015 12330 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.569587 12330 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.569934 12330 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.570216 12326 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.570771 12326 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.577185 11790 catalog_manager.cc:5526] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:51.582180 12347 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:51.579275 11790 catalog_manager.cc:5526] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:51.584574 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.601522 12039 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.604643 11962 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:51.601143 11890 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:51.615263 12038 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:51.616613 11856 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.617537 12084 tablet_bootstrap.cc:654] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.617668 12123 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.619575 12006 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:51.624610 12123 raft_consensus.cc:695] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:51.626230 12150 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.625958 12123 consensus_queue.cc:237] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.627097 12150 raft_consensus.cc:695] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:51.624555 12322 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.628224 12150 consensus_queue.cc:237] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.629338 12322 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.576745 12345 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.633401 12082 raft_consensus.cc:357] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.634279 12082 raft_consensus.cc:383] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.634534 12082 raft_consensus.cc:738] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.634927 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:51.636492 12006 leader_election.cc:343] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.637292 12128 consensus_queue.cc:1035] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:51.637589 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:51.640264 12083 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:51.640741 12083 ts_tablet_manager.cc:1397] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.152s user 0.010s sys 0.014s
I20250113 08:57:51.635107 12082 consensus_queue.cc:260] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:51.641494 11932 leader_election.cc:343] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.644132 12083 raft_consensus.cc:357] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.644448 12322 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.644932 12083 raft_consensus.cc:383] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.645226 12083 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.645944 12083 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.638373 12330 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.650462 12345 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.651537 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.653446 12326 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.652103 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.653558 12040 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.655300 12082 ts_tablet_manager.cc:1428] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.109s user 0.007s sys 0.004s
I20250113 08:57:51.656853 12082 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
W20250113 08:57:51.663218 12007 leader_election.cc:343] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.666539 12007 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.665517 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.662691 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:51.671154 12039 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:51.673426 11931 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:51.675379 12006 leader_election.cc:343] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.680670 12123 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.681125 12123 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.686757 12123 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.679286 12330 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.679400 12226 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:51.680224 12345 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.698038 12083 ts_tablet_manager.cc:1428] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.057s user 0.001s sys 0.012s
I20250113 08:57:51.665676 12347 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20250113 08:57:51.687207 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.699738 12083 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:51.699901 12042 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.697587 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.701618 12041 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
W20250113 08:57:51.688038 12007 leader_election.cc:343] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.692914 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:51.705629 11856 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.702860 12345 raft_consensus.cc:695] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:51.707626 12330 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.709579 12330 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.709957 12330 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:51.710985 11858 leader_election.cc:343] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.706305 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:51.714861 11858 leader_election.cc:343] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:51.715378 12345 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.718501 12084 tablet_bootstrap.cc:492] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:51.718873 12084 ts_tablet_manager.cc:1397] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.210s user 0.012s sys 0.006s
I20250113 08:57:51.721344 12040 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:51.722571 11931 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.729194 12084 raft_consensus.cc:357] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.729983 12084 raft_consensus.cc:383] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:51.730249 12084 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:51.705199 12007 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:51.740213 12347 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.741420 12084 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:51.746747 11858 consensus_peers.cc:487] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:51.751520 12395 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:51.766328 12082 tablet_bootstrap.cc:654] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.758533 12347 raft_consensus.cc:695] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:51.775919 12347 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:51.757980 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 69217.9us
I20250113 08:57:51.788147 12252 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.785630 12330 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.813373 12083 tablet_bootstrap.cc:654] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:51.814903 12123 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.815517 12123 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.818228 12084 ts_tablet_manager.cc:1428] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.099s user 0.012s sys 0.011s
I20250113 08:57:51.819010 12084 tablet_bootstrap.cc:492] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:51.820081 12123 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.799376 12252 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.821923 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.822587 12038 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:51.825727 11856 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.826395 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:51.826862 12252 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.827205 12040 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:51.827519 12252 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:57:51.804251 12042 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:51.828492 11856 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:51.830772 12350 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:51.832937 12350 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.835074 11856 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:51.827973 12252 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:51.836822 12123 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:51.842906 12123 raft_consensus.cc:695] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:51.833372 12350 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.844024 12123 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.851727 12330 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.849540 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.853420 11963 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:51.853528 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:51.855834 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:51.856463 12041 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:51.864262 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:51.846499 12252 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:51.945751 12345 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.946604 12345 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:51.958559 12391 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:51.973690 12226 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:51.974532 12226 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:51.978092 12007 consensus_peers.cc:487] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:51.978140 12226 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:51.990859 12041 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:52.001207 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:52.017726 12007 leader_election.cc:343] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:52.019169 12345 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:51.964862 12350 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.026790 12350 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:52.040613 12082 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:52.090685 12082 ts_tablet_manager.cc:1397] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.434s user 0.033s sys 0.016s
I20250113 08:57:52.110586 12252 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.164470 12083 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:52.165012 12083 ts_tablet_manager.cc:1397] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.466s user 0.011s sys 0.004s
I20250113 08:57:52.163410 12082 raft_consensus.cc:357] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.166518 12082 raft_consensus.cc:383] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:52.166868 12082 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:52.167593 12082 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.169482 12083 raft_consensus.cc:357] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.170383 12083 raft_consensus.cc:383] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:52.170612 12083 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:52.157810 12252 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:52.179076 12082 ts_tablet_manager.cc:1428] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.088s user 0.008s sys 0.000s
I20250113 08:57:52.180421 12082 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:52.181655 12083 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.182883 12084 tablet_bootstrap.cc:654] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:52.184689 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:52.193883 12041 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:52.194382 12083 ts_tablet_manager.cc:1428] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.029s user 0.006s sys 0.003s
I20250113 08:57:52.195385 12083 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:52.248660 11791 catalog_manager.cc:5526] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:52.504889 11790 catalog_manager.cc:5526] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:52.512594 12415 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:52.514072 12415 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.517234 12415 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:52.588852 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:52.597807 12350 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:52.598150 11858 leader_election.cc:343] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:52.598799 12350 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:52.600291 11858 leader_election.cc:343] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:52.602005 12350 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:52.606828 12084 tablet_bootstrap.cc:492] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:52.607882 12084 ts_tablet_manager.cc:1397] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.789s user 0.027s sys 0.004s
I20250113 08:57:52.616946 12084 raft_consensus.cc:357] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.618175 12084 raft_consensus.cc:383] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:52.618602 12084 raft_consensus.cc:738] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:52.619369 12084 consensus_queue.cc:260] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.624469 11856 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:52.626052 12402 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:52.628643 12415 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:52.629348 12415 raft_consensus.cc:695] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:52.630980 12415 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.635715 12084 ts_tablet_manager.cc:1428] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.027s user 0.005s sys 0.005s
I20250113 08:57:52.636081 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:52.638866 11960 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:52.639755 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:52.652112 12084 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:52.647403 12414 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:52.663327 12084 tablet_bootstrap.cc:654] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:52.663395 11858 leader_election.cc:343] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:52.682044 12410 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:52.682991 12410 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.686606 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:52.696645 12399 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:52.697269 12399 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:52.697750 12399 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:52.698057 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:52.697959 12322 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:52.698977 11965 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:57:52.703761 12084 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:52.704315 12084 ts_tablet_manager.cc:1397] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.000s
I20250113 08:57:52.704438 12082 tablet_bootstrap.cc:700] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Time spent opening tablet: real 0.446s user 0.003s sys 0.003s
I20250113 08:57:52.706249 12082 tablet_bootstrap.cc:654] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:52.707008 12410 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:52.711074 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:52.711858 11962 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:52.720253 11858 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:52.733023 12350 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:57:52.749469 12350 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:52.768597 12084 raft_consensus.cc:357] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.771114 12084 raft_consensus.cc:383] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:52.771510 12084 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:52.772634 12084 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.778942 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:52.779945 11967 raft_consensus.cc:2388] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:57:52.770128 12350 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:52.786343 12084 ts_tablet_manager.cc:1428] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.082s user 0.010s sys 0.001s
I20250113 08:57:52.818753 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:52.819612 12042 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:52.825531 11962 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:57:52.827797 11858 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:52.829147 12414 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:52.829849 12414 raft_consensus.cc:695] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:52.831434 12042 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:52.831284 12414 consensus_queue.cc:237] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:52.838096 12008 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.008s sys 0.013s
I20250113 08:57:52.843611 12083 tablet_bootstrap.cc:654] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:52.850768 12084 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:52.851368 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:52.852144 12042 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:57:52.850303 12008 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.010s sys 0.015s
W20250113 08:57:52.817138 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 81890.4us
W20250113 08:57:52.880846 12006 leader_election.cc:343] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:52.859794 12350 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:52.879225 12007 consensus_peers.cc:487] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:57:52.885222 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 90612.3us
I20250113 08:57:52.888175 12042 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:57:52.894685 12402 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:52.895388 12402 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.910085 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:52.911492 12399 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.882506 12006 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:52.942461 12424 log.cc:926] Time spent T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.158s user 0.003s sys 0.000s
I20250113 08:57:52.943280 11890 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:52.947580 12093 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:52.957711 12093 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:52.959429 11790 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44358 (request call id 92) took 1214 ms. Trace:
W20250113 08:57:52.943403 11858 consensus_peers.cc:487] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:57:52.947912 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 67153.9us
I20250113 08:57:52.953449 12326 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.081s
I20250113 08:57:52.948951 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:52.971998 12042 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:52.951563 12150 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:52.964525 12402 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:52.978475 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:52.962581 12350 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:52.962911 12399 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:52.987232 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:52.988766 12408 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250113 08:57:52.989904 11791 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60016 (request call id 101) took 1236 ms. Trace:
I20250113 08:57:52.982540 11965 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:57:52.990270 11791 rpcz_store.cc:276] 0113 08:57:51.753329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:57:51.753442 (+ 113us) service_pool.cc:225] Handling call
0113 08:57:52.989713 (+1236271us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:57:52.437831 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:57:52.438396 (+ 565us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:57:52.438420 (+ 24us) write_op.cc:435] Acquired schema lock
0113 08:57:52.438432 (+ 12us) tablet.cc:659] Decoding operations
0113 08:57:52.438950 (+ 518us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:57:52.439046 (+ 96us) write_op.cc:641] Partition lock acquired for write op
0113 08:57:52.439086 (+ 40us) tablet.cc:682] Acquiring locks for 1 operations
0113 08:57:52.439299 (+ 213us) tablet.cc:698] Row locks acquired
0113 08:57:52.439321 (+ 22us) write_op.cc:260] PREPARE: finished
0113 08:57:52.439508 (+ 187us) write_op.cc:270] Start()
0113 08:57:52.439628 (+ 120us) write_op.cc:276] Timestamp: P: 1736758672439453 usec, L: 0
0113 08:57:52.439654 (+ 26us) op_driver.cc:348] REPLICATION: starting
0113 08:57:52.440426 (+ 772us) log.cc:844] Serialized 543 byte log entry
0113 08:57:52.822105 (+381679us) op_driver.cc:464] REPLICATION: finished
0113 08:57:52.845621 (+ 23516us) write_op.cc:301] APPLY: starting
0113 08:57:52.846191 (+ 570us) tablet.cc:1364] starting BulkCheckPresence
0113 08:57:52.868083 (+ 21892us) tablet.cc:1367] finished BulkCheckPresence
0113 08:57:52.877231 (+ 9148us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:57:52.877703 (+ 472us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:57:52.878112 (+ 409us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0113 08:57:52.878398 (+ 286us) write_op.cc:312] APPLY: finished
0113 08:57:52.915348 (+ 36950us) log.cc:844] Serialized 22 byte log entry
0113 08:57:52.915670 (+ 322us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:57:52.915816 (+ 146us) write_op.cc:454] Released schema lock
0113 08:57:52.915897 (+ 81us) write_op.cc:341] FINISH: Updating metrics
0113 08:57:52.916456 (+ 559us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:57:52.916498 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":425087,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":23155,"apply.run_cpu_time_us":3419,"apply.run_wall_time_us":70740,"num_ops":1,"prepare.queue_time_us":184599,"prepare.run_cpu_time_us":2770,"prepare.run_wall_time_us":2943,"replication_time_us":376243,"thread_start_us":208111,"threads_started":2}]]}
I20250113 08:57:52.959973 11790 rpcz_store.cc:276] 0113 08:57:51.745597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:57:51.745768 (+ 171us) service_pool.cc:225] Handling call
0113 08:57:52.959319 (+1213551us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:57:52.514916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:57:52.515478 (+ 562us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:57:52.515519 (+ 41us) write_op.cc:435] Acquired schema lock
0113 08:57:52.515549 (+ 30us) tablet.cc:659] Decoding operations
0113 08:57:52.516180 (+ 631us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:57:52.516298 (+ 118us) write_op.cc:641] Partition lock acquired for write op
0113 08:57:52.516337 (+ 39us) tablet.cc:682] Acquiring locks for 1 operations
0113 08:57:52.516719 (+ 382us) tablet.cc:698] Row locks acquired
0113 08:57:52.516746 (+ 27us) write_op.cc:260] PREPARE: finished
0113 08:57:52.517054 (+ 308us) write_op.cc:270] Start()
0113 08:57:52.517204 (+ 150us) write_op.cc:276] Timestamp: P: 1736758672516926 usec, L: 0
0113 08:57:52.517235 (+ 31us) op_driver.cc:348] REPLICATION: starting
0113 08:57:52.518618 (+ 1383us) log.cc:844] Serialized 543 byte log entry
0113 08:57:52.851553 (+332935us) op_driver.cc:464] REPLICATION: finished
0113 08:57:52.916985 (+ 65432us) write_op.cc:301] APPLY: starting
0113 08:57:52.917073 (+ 88us) tablet.cc:1364] starting BulkCheckPresence
0113 08:57:52.917203 (+ 130us) tablet.cc:1367] finished BulkCheckPresence
0113 08:57:52.917399 (+ 196us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:57:52.917704 (+ 305us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:57:52.917944 (+ 240us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0113 08:57:52.922334 (+ 4390us) write_op.cc:312] APPLY: finished
0113 08:57:52.922602 (+ 268us) log.cc:844] Serialized 22 byte log entry
0113 08:57:52.922858 (+ 256us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:57:52.922972 (+ 114us) write_op.cc:454] Released schema lock
0113 08:57:52.923030 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0113 08:57:52.923474 (+ 444us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:57:52.923516 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":32351,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":65204,"apply.run_cpu_time_us":2110,"apply.run_wall_time_us":6412,"num_ops":1,"prepare.queue_time_us":360,"prepare.run_cpu_time_us":3893,"prepare.run_wall_time_us":4152,"replication_time_us":334170,"thread_start_us":18164,"threads_started":1}]]}
I20250113 08:57:52.999208 12093 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:53.010716 12407 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:53.011389 12407 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.016817 12326 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:53.017455 12326 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.017879 12407 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:53.016470 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:53.024060 11888 raft_consensus.cc:2363] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:57:53.022897 12082 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:53.024767 12082 ts_tablet_manager.cc:1397] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.845s user 0.027s sys 0.040s
I20250113 08:57:53.030341 12326 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:53.033491 12326 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:53.044636 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:53.042214 11790 catalog_manager.cc:5526] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:53.047489 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:53.047614 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:53.050535 12084 tablet_bootstrap.cc:654] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:53.049544 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:53.051172 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:53.041957 12326 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:53.054507 12007 leader_election.cc:343] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:53.057853 11932 leader_election.cc:343] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:53.059901 12326 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:53.060918 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:53.057219 11790 catalog_manager.cc:5526] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:53.064885 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:53.085175 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:53.089118 12041 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 3
I20250113 08:57:53.086000 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:53.081239 12082 raft_consensus.cc:357] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.128768 12082 raft_consensus.cc:383] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:53.129386 12082 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:53.150193 12083 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:53.153020 12083 ts_tablet_manager.cc:1397] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.958s user 0.032s sys 0.012s
I20250113 08:57:53.154156 11856 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:53.162801 12345 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.054s
I20250113 08:57:53.130751 11790 catalog_manager.cc:5526] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:53.172475 12006 leader_election.cc:400] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:57:53.172864 12006 leader_election.cc:403] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:57:53.139583 12082 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.181478 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:53.183802 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:53.186916 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:53.191434 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:53.199321 12437 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
W20250113 08:57:53.199187 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45360us
I20250113 08:57:53.174585 12415 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:53.201551 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 48360us
I20250113 08:57:53.214509 12084 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:53.229478 12037 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:53.231899 12415 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.215070 12084 ts_tablet_manager.cc:1397] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.401s user 0.027s sys 0.006s
I20250113 08:57:53.224128 12437 raft_consensus.cc:695] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:57:53.248468 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 136246us
W20250113 08:57:53.260627 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 47367.1us
W20250113 08:57:53.250299 11932 consensus_peers.cc:487] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Couldn't send request to peer 6ea00811b7d44b49b872ac5c03b91487. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:53.285991 12006 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:53.294392 12084 raft_consensus.cc:357] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.305768 12084 raft_consensus.cc:383] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:53.308262 12084 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:53.288736 11931 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:53.329602 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 45108.2us
I20250113 08:57:53.332343 11931 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:53.339054 12084 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:53.345269 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 52993.6us
W20250113 08:57:53.285809 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 57814.3us
I20250113 08:57:53.353055 11856 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:53.333724 12399 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:53.370335 11965 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:57:53.371403 12437 consensus_queue.cc:237] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.372742 12082 ts_tablet_manager.cc:1428] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.348s user 0.012s sys 0.007s
I20250113 08:57:53.383428 12083 raft_consensus.cc:357] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.386363 12083 raft_consensus.cc:383] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:53.386747 12083 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:53.387516 12083 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:53.383353 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 114025us
W20250113 08:57:53.398077 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 47027.7us
W20250113 08:57:53.400339 11932 leader_election.cc:343] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:53.401527 12084 ts_tablet_manager.cc:1428] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.163s user 0.032s sys 0.000s
W20250113 08:57:53.403473 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 71515.5us
I20250113 08:57:53.406610 11858 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:53.353719 12420 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20250113 08:57:53.411351 12252 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:53.412081 12084 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:53.412685 12083 ts_tablet_manager.cc:1428] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.259s user 0.007s sys 0.008s
I20250113 08:57:53.414290 12395 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:53.416381 12093 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 08:57:53.422457 11858 consensus_peers.cc:487] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:53.426878 12037 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:53.427768 12083 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:53.431185 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:53.405623 12402 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:53.426465 12414 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
W20250113 08:57:53.443594 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 85566.3us
I20250113 08:57:53.439322 12345 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
W20250113 08:57:53.445359 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:53.445525 12345 raft_consensus.cc:695] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:57:53.439365 11822 debug-util.cc:398] Leaking SignalData structure 0x7b080003d180 after lost signal to thread 10705
I20250113 08:57:53.467130 12082 tablet_bootstrap.cc:492] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:53.480208 12415 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:53.482604 12083 tablet_bootstrap.cc:654] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:53.486472 12414 raft_consensus.cc:695] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:53.493491 12414 consensus_queue.cc:237] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.495944 12006 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:53.496824 12408 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:57:53.498060 12006 leader_election.cc:343] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:53.500250 12006 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:53.498561 12345 consensus_queue.cc:237] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.517174 11892 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:53.516762 12408 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:53.520681 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 62642us
I20250113 08:57:53.530696 12415 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.136s
W20250113 08:57:53.674880 12041 consensus_meta.cc:287] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.532s user 0.014s sys 0.003s
W20250113 08:57:53.676080 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 178043us
I20250113 08:57:53.688270 12084 tablet_bootstrap.cc:654] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:53.710332 12451 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:53.678468 12041 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:57:53.711185 12451 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.712426 12150 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
W20250113 08:57:53.676308 11822 debug-util.cc:398] Leaking SignalData structure 0x7b0800036ca0 after lost signal to thread 11825
W20250113 08:57:53.722841 12455 log.cc:926] Time spent T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
W20250113 08:57:53.727969 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 272220us
I20250113 08:57:53.838601 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:57:53.840056 11822 debug-util.cc:398] Leaking SignalData structure 0x7b0800298b80 after lost signal to thread 11917
I20250113 08:57:53.862394 12456 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20250113 08:57:53.866838 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08002993c0 after lost signal to thread 11992
I20250113 08:57:53.868232 11962 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:53.882992 12463 log.cc:926] Time spent T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20250113 08:57:53.884672 12407 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:53.885259 12407 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.887331 12407 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:57:53.880069 11822 debug-util.cc:398] Leaking SignalData structure 0x7b0800217b80 after lost signal to thread 12066
I20250113 08:57:53.901441 12082 tablet_bootstrap.cc:654] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:53.913609 11791 catalog_manager.cc:5526] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:53.918481 12464 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:53.919071 12084 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:53.919756 12084 ts_tablet_manager.cc:1397] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.516s user 0.025s sys 0.012s
I20250113 08:57:53.919260 12464 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.924094 12464 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:53.924888 12084 raft_consensus.cc:357] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.925787 12084 raft_consensus.cc:383] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:53.926168 12084 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:53.924446 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:53.898165 12395 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:53.933547 12084 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.936427 12395 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.939322 12083 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:53.939797 12083 ts_tablet_manager.cc:1397] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.512s user 0.026s sys 0.014s
I20250113 08:57:53.941128 12395 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:53.938673 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:53.960996 12083 raft_consensus.cc:357] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.961776 12083 raft_consensus.cc:383] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:53.962072 12083 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:53.972328 12084 ts_tablet_manager.cc:1428] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.052s user 0.009s sys 0.000s
I20250113 08:57:53.972807 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:53.974155 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:53.977773 12007 leader_election.cc:343] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:53.965011 12083 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.984357 12226 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:53.985275 12083 ts_tablet_manager.cc:1428] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20250113 08:57:53.984869 12226 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:53.987917 12226 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:53.988621 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:54.037971 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:54.042871 12035 raft_consensus.cc:1270] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250113 08:57:54.048063 12451 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:54.018230 11967 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:54.049568 11963 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:54.049721 11968 raft_consensus.cc:1270] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250113 08:57:54.052901 11967 raft_consensus.cc:1270] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250113 08:57:54.053396 12084 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:54.062968 12083 tablet_bootstrap.cc:492] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
W20250113 08:57:54.065042 12474 log.cc:926] Time spent T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
W20250113 08:57:54.065495 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 87993.7us
I20250113 08:57:54.066480 11932 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:54.067380 11858 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:54.081885 12402 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:54.123654 12007 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:54.131062 12420 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:54.131392 12252 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:57:54.132119 12252 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.133442 12461 consensus_queue.cc:1035] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
W20250113 08:57:54.165943 12007 leader_election.cc:343] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:54.127121 12437 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20250113 08:57:54.166923 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:54.159078 12252 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:54.156023 12415 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.017s
I20250113 08:57:54.142200 12449 consensus_queue.cc:1035] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20250113 08:57:54.213968 12458 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:57:54.227051 12039 raft_consensus.cc:1270] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250113 08:57:54.224668 11961 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:54.235512 12084 tablet_bootstrap.cc:654] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:54.239449 12082 tablet_bootstrap.cc:492] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:54.239917 12082 ts_tablet_manager.cc:1397] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.786s user 0.028s sys 0.070s
I20250113 08:57:54.238291 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:54.225021 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:54.252842 12082 raft_consensus.cc:357] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.259738 12082 raft_consensus.cc:383] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:54.259979 12082 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:54.257604 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:54.260888 12465 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:57:54.269799 12083 tablet_bootstrap.cc:654] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:54.260438 12082 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.258395 12414 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20250113 08:57:54.268170 12454 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:54.243713 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:54.292042 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:54.269481 12465 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.308075 12082 ts_tablet_manager.cc:1428] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.068s user 0.005s sys 0.001s
I20250113 08:57:54.285554 12454 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:54.341187 12485 log.cc:926] Time spent T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250113 08:57:54.343421 12465 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:54.354272 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:54.355238 11888 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:54.355947 12454 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:54.357966 12458 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:54.358456 12458 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:57:54.363538 12007 leader_election.cc:343] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:54.372251 12093 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
W20250113 08:57:54.385416 11858 consensus_peers.cc:487] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:57:54.386015 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 242959us
I20250113 08:57:54.390049 12006 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:54.401414 12006 leader_election.cc:343] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:54.401894 12006 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:54.407313 11858 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:54.408107 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:54.404543 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:54.416191 12082 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:54.418475 12006 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
W20250113 08:57:54.413162 12007 leader_election.cc:343] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:54.405853 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:54.424911 12450 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:54.433342 12460 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:54.433729 12457 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:54.428812 12395 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:54.450353 12395 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:54.461019 12395 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:54.445411 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:54.463423 11932 consensus_peers.cc:487] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Couldn't send request to peer 6ea00811b7d44b49b872ac5c03b91487. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:57:54.478734 12488 log.cc:926] Time spent T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250113 08:57:54.494091 11856 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:54.513737 12437 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:54.514031 12093 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:57:54.545284 12006 leader_election.cc:343] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:54.545871 12006 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:54.570881 12392 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:54.571521 12392 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.603729 12082 tablet_bootstrap.cc:654] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:54.587070 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:54.552855 12093 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.594895 12467 raft_consensus.cc:2743] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:54.583935 12392 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:54.623512 12083 tablet_bootstrap.cc:492] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:54.624145 12083 ts_tablet_manager.cc:1397] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.562s user 0.021s sys 0.019s
W20250113 08:57:54.628460 12007 leader_election.cc:343] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:54.604596 12458 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.634752 12093 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:54.636382 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:54.648407 12006 leader_election.cc:343] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:54.648910 12006 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:54.657199 12470 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:54.669802 12084 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:54.670369 12084 ts_tablet_manager.cc:1397] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.617s user 0.017s sys 0.015s
I20250113 08:57:54.673072 12458 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:54.675645 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:54.679098 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 143983us
W20250113 08:57:54.693329 12492 log.cc:926] Time spent T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20250113 08:57:54.674247 12042 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:57:54.709961 12084 raft_consensus.cc:357] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.721531 12084 raft_consensus.cc:383] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:54.723798 12084 raft_consensus.cc:738] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:54.695194 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:57:54.731429 12494 log.cc:926] Time spent T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250113 08:57:54.731202 12084 consensus_queue.cc:260] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.738263 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:54.804782 12083 raft_consensus.cc:357] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:54.808200 12083 raft_consensus.cc:383] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:54.808499 12083 raft_consensus.cc:738] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:54.811167 12083 consensus_queue.cc:260] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:54.815019 12491 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.235s user 0.000s sys 0.002s
W20250113 08:57:54.833415 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:54.835021 11932 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:54.836131 12501 log.cc:926] Time spent T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20250113 08:57:54.836913 12402 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:54.837898 12084 ts_tablet_manager.cc:1428] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.167s user 0.010s sys 0.003s
W20250113 08:57:54.819586 12504 log.cc:926] Time spent T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20250113 08:57:54.845814 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:54.846196 12084 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:54.831626 12395 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:54.899109 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 73810.4us
W20250113 08:57:54.928045 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 95146.6us
I20250113 08:57:54.981598 12083 ts_tablet_manager.cc:1428] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.357s user 0.009s sys 0.022s
W20250113 08:57:54.959548 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 48520.1us
I20250113 08:57:55.023754 12466 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.024322 12466 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.027055 12006 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:55.040395 12505 log.cc:926] Time spent T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20250113 08:57:55.032919 12395 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:55.035547 12458 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:57:55.048310 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:55.054458 11884 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.057742 11791 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44358 (request call id 93) took 1645 ms. Trace:
W20250113 08:57:55.062523 12499 log.cc:926] Time spent T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20250113 08:57:55.052907 12084 tablet_bootstrap.cc:654] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.066170 12466 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:55.058223 11791 rpcz_store.cc:276] 0113 08:57:53.412033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:57:53.412213 (+ 180us) service_pool.cc:225] Handling call
0113 08:57:55.057676 (+1645463us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:57:54.213677 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:57:54.240687 (+ 27010us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:57:54.241831 (+ 1144us) write_op.cc:435] Acquired schema lock
0113 08:57:54.241859 (+ 28us) tablet.cc:659] Decoding operations
0113 08:57:54.273084 (+ 31225us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:57:54.279520 (+ 6436us) write_op.cc:641] Partition lock acquired for write op
0113 08:57:54.281806 (+ 2286us) tablet.cc:682] Acquiring locks for 1 operations
0113 08:57:54.302632 (+ 20826us) tablet.cc:698] Row locks acquired
0113 08:57:54.302660 (+ 28us) write_op.cc:260] PREPARE: finished
0113 08:57:54.322978 (+ 20318us) write_op.cc:270] Start()
0113 08:57:54.325318 (+ 2340us) write_op.cc:276] Timestamp: P: 1736758674322459 usec, L: 0
0113 08:57:54.325341 (+ 23us) op_driver.cc:348] REPLICATION: starting
0113 08:57:54.325976 (+ 635us) log.cc:844] Serialized 543 byte log entry
0113 08:57:54.440033 (+114057us) op_driver.cc:464] REPLICATION: finished
0113 08:57:54.659009 (+218976us) write_op.cc:301] APPLY: starting
0113 08:57:54.659093 (+ 84us) tablet.cc:1364] starting BulkCheckPresence
0113 08:57:54.659197 (+ 104us) tablet.cc:1367] finished BulkCheckPresence
0113 08:57:54.699178 (+ 39981us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:57:54.777599 (+ 78421us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:57:54.800184 (+ 22585us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0113 08:57:54.802878 (+ 2694us) write_op.cc:312] APPLY: finished
0113 08:57:54.829602 (+ 26724us) log.cc:844] Serialized 22 byte log entry
0113 08:57:54.849334 (+ 19732us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:57:54.854256 (+ 4922us) write_op.cc:454] Released schema lock
0113 08:57:54.861507 (+ 7251us) write_op.cc:341] FINISH: Updating metrics
0113 08:57:54.964477 (+102970us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:57:54.971432 (+ 6955us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":218708,"apply.run_cpu_time_us":6660,"apply.run_wall_time_us":266570,"num_ops":1,"prepare.queue_time_us":238269,"prepare.run_cpu_time_us":3656,"prepare.run_wall_time_us":112648,"replication_time_us":114504,"thread_start_us":421808,"threads_started":2}]]}
I20250113 08:57:55.074997 12467 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.096966 12082 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:55.100443 12082 ts_tablet_manager.cc:1397] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.720s user 0.031s sys 0.054s
I20250113 08:57:55.090991 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:55.148684 11884 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:55.101426 12467 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:55.164011 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 44344.3us
W20250113 08:57:55.307109 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 276110us
I20250113 08:57:55.288527 12082 raft_consensus.cc:357] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.308959 12082 raft_consensus.cc:383] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.309312 12082 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.310601 12082 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.328811 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:55.331735 11932 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:55.334352 12464 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:55.341114 12084 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:55.343147 12084 ts_tablet_manager.cc:1397] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.497s user 0.006s sys 0.019s
I20250113 08:57:55.349594 12006 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:55.351030 12326 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.353772 11966 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.351758 12326 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.358958 12326 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:55.367189 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:55.367448 12467 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:55.355854 12226 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:55.374305 12226 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:55.375459 12226 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.386948 12084 raft_consensus.cc:357] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.387897 12084 raft_consensus.cc:383] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.388121 12084 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.388559 12082 ts_tablet_manager.cc:1428] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.288s user 0.007s sys 0.002s
I20250113 08:57:55.388857 12084 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.395746 12084 ts_tablet_manager.cc:1428] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20250113 08:57:55.400008 12083 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:55.405881 12082 tablet_bootstrap.cc:492] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:55.413180 12084 tablet_bootstrap.cc:492] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:55.418795 12449 consensus_queue.cc:1046] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:55.419952 12449 raft_consensus.cc:3049] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:57:55.420325 12449 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:55.420861 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:55.421103 12449 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.423679 12449 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:55.422797 12082 tablet_bootstrap.cc:654] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.423538 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:55.431555 12084 tablet_bootstrap.cc:654] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.439105 12082 tablet_bootstrap.cc:492] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:55.439697 12082 ts_tablet_manager.cc:1397] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.004s
W20250113 08:57:55.441746 12006 leader_election.cc:343] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.443490 12082 raft_consensus.cc:357] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.446943 12082 raft_consensus.cc:383] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.447186 12082 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
W20250113 08:57:55.447187 12007 leader_election.cc:343] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.447970 12082 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.458482 12082 ts_tablet_manager.cc:1428] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.018s user 0.003s sys 0.002s
I20250113 08:57:55.459658 12082 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:55.459758 11791 catalog_manager.cc:5526] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:55.460640 12084 tablet_bootstrap.cc:492] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:55.461213 12084 ts_tablet_manager.cc:1397] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.010s
I20250113 08:57:55.461797 12083 tablet_bootstrap.cc:654] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.467231 12084 raft_consensus.cc:357] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:55.467480 12006 leader_election.cc:343] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.467865 12084 raft_consensus.cc:383] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.468124 12084 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.469734 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:55.470032 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:55.461934 11791 catalog_manager.cc:5526] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } }
I20250113 08:57:55.474637 12084 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:55.475306 12007 leader_election.cc:343] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.475741 12007 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:55.476430 12450 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:55.476778 12007 leader_election.cc:343] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.477177 12007 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:55.479481 12007 consensus_peers.cc:487] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20250113 08:57:55.480736 12450 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:55.481391 11791 catalog_manager.cc:5526] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 2, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } }
I20250113 08:57:55.488023 11789 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60016 (request call id 102) took 2024 ms. Trace:
I20250113 08:57:55.488647 11789 rpcz_store.cc:276] 0113 08:57:53.449534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:57:53.456251 (+ 6717us) service_pool.cc:225] Handling call
0113 08:57:55.466783 (+2010532us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
I20250113 08:57:55.483892 12082 tablet_bootstrap.cc:654] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.484503 12084 ts_tablet_manager.cc:1428] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250113 08:57:55.500149 12084 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:55.508991 12084 tablet_bootstrap.cc:654] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.512097 12083 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:55.512710 12083 ts_tablet_manager.cc:1397] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.113s user 0.007s sys 0.013s
I20250113 08:57:55.523177 12082 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:55.523825 12082 ts_tablet_manager.cc:1397] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.012s
I20250113 08:57:55.526808 12082 raft_consensus.cc:357] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.530107 12082 raft_consensus.cc:383] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.530719 12082 raft_consensus.cc:738] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.531522 12082 consensus_queue.cc:260] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.532791 12084 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:55.537101 12084 ts_tablet_manager.cc:1397] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.002s
I20250113 08:57:55.540036 12084 raft_consensus.cc:357] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.540781 12084 raft_consensus.cc:383] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.534505 12083 raft_consensus.cc:357] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.541571 12083 raft_consensus.cc:383] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.541924 12083 raft_consensus.cc:738] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.542711 12083 consensus_queue.cc:260] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.545410 12083 ts_tablet_manager.cc:1428] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
I20250113 08:57:55.541049 12084 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.546311 12084 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.548708 12084 ts_tablet_manager.cc:1428] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250113 08:57:55.552234 12084 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:55.549667 12083 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:55.563407 12459 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.563966 12459 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.564550 12083 tablet_bootstrap.cc:654] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.565764 12082 ts_tablet_manager.cc:1428] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.042s user 0.005s sys 0.001s
I20250113 08:57:55.566843 12082 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:55.573809 12459 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:55.574540 12082 tablet_bootstrap.cc:654] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.575039 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:55.577318 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:55.578029 12042 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
W20250113 08:57:55.579464 11858 leader_election.cc:343] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.581853 11856 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:55.582505 12459 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:55.582928 12459 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:55.583316 12459 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.586349 11790 catalog_manager.cc:5526] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:55.593443 12084 tablet_bootstrap.cc:654] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.597151 12083 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:55.598501 12083 ts_tablet_manager.cc:1397] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.000s
I20250113 08:57:55.608387 11891 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.623042 12083 raft_consensus.cc:357] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.624053 12083 raft_consensus.cc:383] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.624349 12083 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.624866 12083 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.627326 12083 ts_tablet_manager.cc:1428] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250113 08:57:55.628410 12083 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:55.630251 11789 catalog_manager.cc:5526] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:55.633683 12226 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.634560 12226 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.636516 12226 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:55.638275 12459 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.642542 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:55.654489 12083 tablet_bootstrap.cc:654] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:55.655201 12006 leader_election.cc:343] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.655361 12459 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:55.656950 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:55.644317 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:55.646831 12456 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.659957 12456 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.670639 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:55.671329 12041 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.672066 12456 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:57:55.661684 12007 leader_election.cc:343] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.672767 12007 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:55.673545 11858 leader_election.cc:343] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.679193 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:55.680157 11962 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:55.681330 12082 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:55.681841 12082 ts_tablet_manager.cc:1397] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.115s user 0.020s sys 0.002s
I20250113 08:57:55.681582 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:55.684206 11858 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:55.692808 12456 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:55.696655 12456 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:55.697144 12456 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.689922 12226 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:55.682477 12042 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:55.713830 12464 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.714407 12464 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.716553 12464 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:55.729831 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:55.730523 12038 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:55.731069 12041 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:55.733453 11931 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:55.740281 11856 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:55.741655 12449 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:55.742152 12449 raft_consensus.cc:695] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:55.742976 12449 consensus_queue.cc:237] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.750473 12466 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:55.750972 12466 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:55.751307 12466 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.718830 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:55.758737 12082 raft_consensus.cc:357] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.759752 12464 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.760659 12082 raft_consensus.cc:383] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.762526 12464 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.765182 12082 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.769975 12464 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:57:55.773355 11932 leader_election.cc:343] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.780130 12082 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.783741 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:55.789793 11887 raft_consensus.cc:2388] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:55.787460 12456 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.834992 12083 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:55.837209 12083 ts_tablet_manager.cc:1397] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.209s user 0.013s sys 0.018s
I20250113 08:57:55.854509 12082 ts_tablet_manager.cc:1428] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.172s user 0.011s sys 0.000s
I20250113 08:57:55.860736 11963 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.877015 12082 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:55.850040 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:55.895977 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:55.896013 11931 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:55.896759 12084 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:55.896814 11966 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.897821 12084 ts_tablet_manager.cc:1397] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.346s user 0.020s sys 0.055s
I20250113 08:57:55.910038 12083 raft_consensus.cc:357] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.910741 12083 raft_consensus.cc:383] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.911000 12083 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.911653 12083 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.912686 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:55.913329 12038 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:55.903990 12456 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:55.917138 12459 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.933318 11966 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:55.905040 12466 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.942329 12083 ts_tablet_manager.cc:1428] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.105s user 0.005s sys 0.003s
I20250113 08:57:55.945650 12084 raft_consensus.cc:357] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.952936 12084 raft_consensus.cc:383] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:55.953228 12084 raft_consensus.cc:738] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:55.921355 12459 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.955950 12084 consensus_queue.cc:260] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.966320 12084 ts_tablet_manager.cc:1428] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
W20250113 08:57:55.976704 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:55.978628 12392 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:55.979188 12392 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:55.983075 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:55.985589 12007 leader_election.cc:343] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:55.987205 12082 tablet_bootstrap.cc:654] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:55.993703 12084 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:55.988718 12392 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:56.012037 12459 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.022049 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:56.022959 12033 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.025550 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:57:56.032141 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 115813us
I20250113 08:57:56.019090 12466 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.035840 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:56.038743 11858 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:56.039899 11858 leader_election.cc:343] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.041440 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:56.044380 12083 tablet_bootstrap.cc:492] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:56.036911 12464 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:56.048702 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.042536 12449 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:56.050221 12449 raft_consensus.cc:695] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:57:56.039570 11932 leader_election.cc:343] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.051061 12449 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.052762 12450 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 08:57:56.062759 11856 leader_election.cc:343] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.063256 11856 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:56.064518 12459 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:56.067018 12006 leader_election.cc:343] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.067417 12006 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:56.069183 12460 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:56.069823 12460 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:56.079093 12466 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.085125 12466 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.088594 12466 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.102250 12038 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:56.096850 12033 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:56.103876 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.118160 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:56.119971 12083 tablet_bootstrap.cc:654] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.121856 11963 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.121888 11932 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:56.123224 12084 tablet_bootstrap.cc:654] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.135155 11931 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:56.141619 12347 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:56.142191 12347 raft_consensus.cc:695] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:56.143923 12347 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.176986 12082 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:56.177604 12082 ts_tablet_manager.cc:1397] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.301s user 0.027s sys 0.006s
I20250113 08:57:56.202860 12082 raft_consensus.cc:357] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.205220 12082 raft_consensus.cc:383] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.205608 12082 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.204433 12461 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:56.208334 11789 catalog_manager.cc:5526] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:56.212153 12082 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.211807 11789 catalog_manager.cc:5526] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:56.214988 12084 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:56.215534 12084 ts_tablet_manager.cc:1397] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.225s user 0.025s sys 0.010s
I20250113 08:57:56.218456 12082 ts_tablet_manager.cc:1428] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.041s user 0.004s sys 0.003s
I20250113 08:57:56.226362 12082 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:56.220532 12084 raft_consensus.cc:357] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.229372 12084 raft_consensus.cc:383] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.200707 11790 catalog_manager.cc:5526] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:56.230948 12451 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.233455 12084 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.231778 12451 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.235571 12084 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.248008 12083 tablet_bootstrap.cc:492] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:56.248713 12083 ts_tablet_manager.cc:1397] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.211s user 0.010s sys 0.009s
I20250113 08:57:56.268777 12451 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:56.271392 12083 raft_consensus.cc:357] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.272286 12083 raft_consensus.cc:383] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.272706 12083 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.274916 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:56.273569 12083 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:56.278090 12006 leader_election.cc:343] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.278496 12083 ts_tablet_manager.cc:1428] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20250113 08:57:56.279875 12083 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:56.286135 12084 ts_tablet_manager.cc:1428] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.070s user 0.006s sys 0.001s
I20250113 08:57:56.287541 12084 tablet_bootstrap.cc:492] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:56.295866 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:56.295156 12466 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:56.294687 12461 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.299011 12461 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.301527 12461 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.303424 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.307669 12034 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:57:56.312870 12083 tablet_bootstrap.cc:654] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.313634 12082 tablet_bootstrap.cc:654] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
W20250113 08:57:56.318059 12007 leader_election.cc:343] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.323323 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:56.326645 12084 tablet_bootstrap.cc:654] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.327571 12007 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:56.329517 12470 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250113 08:57:56.331091 11858 consensus_peers.cc:487] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:56.331787 11858 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:56.332531 12542 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.335918 12466 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:56.336488 12466 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.338869 12466 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.339476 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:56.340409 11892 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:57:56.343923 11932 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:56.345182 12466 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:57:56.345690 12466 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:56.346272 12466 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:57:56.347626 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.348629 12033 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:56.374549 12542 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:56.376541 12466 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.383533 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:56.396878 11891 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:57:56.398486 11932 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:56.399577 12556 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:57:56.400086 12556 raft_consensus.cc:695] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:56.400712 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:56.400919 12556 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.401669 12033 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:56.406333 12083 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:56.406841 12083 ts_tablet_manager.cc:1397] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.127s user 0.017s sys 0.016s
I20250113 08:57:56.408757 12084 tablet_bootstrap.cc:492] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:56.409513 12084 ts_tablet_manager.cc:1397] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.122s user 0.014s sys 0.000s
I20250113 08:57:56.409540 12466 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.412392 12084 raft_consensus.cc:357] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.413152 12084 raft_consensus.cc:383] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.413463 12084 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.414162 12084 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.416435 12084 ts_tablet_manager.cc:1428] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250113 08:57:56.416941 12083 raft_consensus.cc:357] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.417675 12083 raft_consensus.cc:383] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.417886 12083 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.418449 12083 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.426041 12084 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:56.437402 12083 ts_tablet_manager.cc:1428] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20250113 08:57:56.446897 11789 catalog_manager.cc:5526] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:56.456327 12082 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:56.456780 12082 ts_tablet_manager.cc:1397] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.231s user 0.004s sys 0.020s
I20250113 08:57:56.460103 12083 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:56.462044 12082 raft_consensus.cc:357] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.462798 12082 raft_consensus.cc:383] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.463073 12082 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.469400 12082 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.471768 12082 ts_tablet_manager.cc:1428] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20250113 08:57:56.472074 12084 tablet_bootstrap.cc:654] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.472728 12082 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:56.484858 12082 tablet_bootstrap.cc:654] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.492709 12082 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:56.493221 12082 ts_tablet_manager.cc:1397] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20250113 08:57:56.496582 12082 raft_consensus.cc:357] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.497349 12082 raft_consensus.cc:383] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.497632 12082 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.498348 12082 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.500603 12082 ts_tablet_manager.cc:1428] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20250113 08:57:56.500661 12083 tablet_bootstrap.cc:654] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.502632 12082 tablet_bootstrap.cc:492] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:56.510546 12082 tablet_bootstrap.cc:654] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.511824 12083 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:56.512367 12083 ts_tablet_manager.cc:1397] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
I20250113 08:57:56.515144 12083 raft_consensus.cc:357] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.515892 12083 raft_consensus.cc:383] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.516187 12083 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.516955 12083 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.525024 12083 ts_tablet_manager.cc:1428] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250113 08:57:56.526091 12083 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Bootstrap starting.
I20250113 08:57:56.539117 12084 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:56.539746 12084 ts_tablet_manager.cc:1397] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.116s user 0.010s sys 0.003s
I20250113 08:57:56.542232 12542 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:56.542658 12083 tablet_bootstrap.cc:654] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.542578 12084 raft_consensus.cc:357] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.543637 12084 raft_consensus.cc:383] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.543951 12084 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.546780 12082 tablet_bootstrap.cc:492] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:56.547433 12082 ts_tablet_manager.cc:1397] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.045s user 0.018s sys 0.008s
I20250113 08:57:56.558161 12082 raft_consensus.cc:357] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.558972 12082 raft_consensus.cc:383] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.559311 12082 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.559991 12082 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.563030 12082 ts_tablet_manager.cc:1428] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250113 08:57:56.564437 12459 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:56.544761 12084 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.573689 12083 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: No bootstrap required, opened a new log
I20250113 08:57:56.574184 12083 ts_tablet_manager.cc:1397] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.011s
I20250113 08:57:56.575138 12084 ts_tablet_manager.cc:1428] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250113 08:57:56.577015 12084 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:56.590292 12084 tablet_bootstrap.cc:654] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.592720 12465 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.593302 12465 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.594301 12083 raft_consensus.cc:357] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.595119 12083 raft_consensus.cc:383] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.595398 12083 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.595463 12465 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.598582 12083 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.598943 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.599643 12033 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:56.600325 12083 ts_tablet_manager.cc:1428] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent starting tablet: real 0.026s user 0.003s sys 0.002s
I20250113 08:57:56.602487 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:56.605058 12556 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.605628 12556 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.609198 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.610049 12033 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:56.610895 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:56.611199 11931 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:56.612588 11891 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:56.612720 12466 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250113 08:57:56.606348 11858 leader_election.cc:343] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.608070 12556 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.613987 12466 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:56.614460 12466 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.614688 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:56.615993 12465 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:56.616384 12465 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:56.616721 12465 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.622133 12466 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.625701 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:56.626358 11891 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.627470 12465 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.630957 12082 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:56.640415 12082 tablet_bootstrap.cc:654] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.640632 11891 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:56.641661 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:56.642102 11932 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:56.643272 12553 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:56.643752 12553 raft_consensus.cc:695] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:56.644774 12553 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.646986 12451 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.647845 12464 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.647512 12451 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.648296 12464 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.651116 12451 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:56.652551 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:56.653983 12006 leader_election.cc:343] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.654470 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:56.654623 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:56.655130 11858 leader_election.cc:343] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:57:56.655705 12007 leader_election.cc:343] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:56.656150 12007 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:56.656580 12466 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.657557 12084 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:56.658103 12084 ts_tablet_manager.cc:1397] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.000s
I20250113 08:57:56.658152 12392 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:56.658080 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:56.658921 12038 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.659436 12464 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.661696 12084 raft_consensus.cc:357] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.662292 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.663949 12084 raft_consensus.cc:383] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.664638 12084 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.665019 11931 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:56.665905 12084 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.666668 12347 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:56.667940 12038 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:56.669590 12084 ts_tablet_manager.cc:1428] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250113 08:57:56.670640 12084 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:56.673674 12082 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:56.674214 12082 ts_tablet_manager.cc:1397] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.003s
I20250113 08:57:56.675887 12465 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.677259 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:56.678193 12033 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.677068 12082 raft_consensus.cc:357] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.678892 12082 raft_consensus.cc:383] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.679298 12082 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.679988 12082 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.679392 12464 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.681298 12464 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.682898 12451 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:56.683422 12451 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.683908 12082 ts_tablet_manager.cc:1428] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20250113 08:57:56.686234 12082 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:56.687544 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:56.688275 11967 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:56.689474 12007 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:56.689937 12451 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:56.690630 12392 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:56.691020 12392 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:56.691351 12392 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.692037 12556 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:56.694662 12084 tablet_bootstrap.cc:654] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.700315 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:56.700954 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:56.707574 11891 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:56.707664 11886 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:56.708873 12556 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:56.704111 12464 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.703234 12392 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.718889 12392 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:56.722752 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:56.723618 12034 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:56.726377 11931 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:56.727241 12347 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:56.727650 12347 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:56.728013 12347 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.708909 12082 tablet_bootstrap.cc:654] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:56.748353 12347 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.758160 12033 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
W20250113 08:57:56.795100 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 83456.8us
I20250113 08:57:56.797379 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:56.797978 11967 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.799939 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:56.800606 11892 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.809271 11967 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:56.817123 11790 catalog_manager.cc:5526] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:56.821199 11891 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:56.822978 12553 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:57:56.825760 12007 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:56.826648 12470 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:56.827235 12470 raft_consensus.cc:695] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:56.828557 12470 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.833531 11892 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:56.834257 12082 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:56.834755 12082 ts_tablet_manager.cc:1397] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.149s user 0.015s sys 0.010s
I20250113 08:57:56.839783 12082 raft_consensus.cc:357] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.840955 12082 raft_consensus.cc:383] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:56.849962 12082 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:56.852073 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:56.853108 12347 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:56.856547 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:56.858867 12350 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:56.853200 11892 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.859680 12350 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:56.866621 12350 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.851054 12082 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.895152 11963 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.896202 12082 ts_tablet_manager.cc:1428] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.060s user 0.004s sys 0.003s
I20250113 08:57:56.898906 11892 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:56.900691 11932 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:56.901878 12347 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:56.906049 12082 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:56.908285 12347 raft_consensus.cc:695] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:56.911031 12347 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:56.932577 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:56.933782 12038 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:56.953114 12038 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:56.955969 12451 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:57.036893 12577 log.cc:926] Time spent T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.110s user 0.002s sys 0.000s
I20250113 08:57:57.081499 12465 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:57.082082 12465 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.110694 12465 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:57:57.150349 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 44410.6us
I20250113 08:57:57.009050 12451 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.239068 12451 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:57.294354 12038 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:57:57.315028 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:57.341977 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:57.338246 11789 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:57.338593 12581 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:57.316563 12347 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.031s
I20250113 08:57:57.347205 12461 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:57.387887 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:57.388686 12038 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:57.381374 12461 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:57.393985 12584 log.cc:926] Time spent T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250113 08:57:57.377615 12581 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.397091 12084 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:57.399308 12082 tablet_bootstrap.cc:654] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:57.403461 12084 ts_tablet_manager.cc:1397] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.733s user 0.052s sys 0.070s
I20250113 08:57:57.427886 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:57.540941 12461 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:57.592731 12581 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:57.633801 12084 raft_consensus.cc:357] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.634891 12084 raft_consensus.cc:383] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:57.635221 12084 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:57.636160 12084 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.702533 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:57.734520 12084 ts_tablet_manager.cc:1428] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.331s user 0.018s sys 0.009s
I20250113 08:57:57.735925 12084 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:57.754575 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:57.793562 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 94543.2us
W20250113 08:57:57.794585 11858 leader_election.cc:343] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:57.793320 12589 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250113 08:57:57.804634 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:57.804620 12007 leader_election.cc:343] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:57.793751 12464 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:57.808701 12464 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.810990 12464 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:57.813582 12556 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:57.813586 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:57.817984 11893 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:57.822168 12006 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:57.829924 12589 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:57.832922 12082 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:57.833505 12082 ts_tablet_manager.cc:1397] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.928s user 0.035s sys 0.022s
W20250113 08:57:57.848654 12006 leader_election.cc:343] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:57.849330 12006 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:57.871129 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20250113 08:57:57.897699 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 265505us
W20250113 08:57:57.898348 11858 leader_election.cc:343] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:57.862675 12589 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:57.862825 12347 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
I20250113 08:57:57.886703 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:57.919142 12039 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:57.922199 12451 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:57.933777 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 43691.4us
W20250113 08:57:57.932778 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 81500us
I20250113 08:57:57.942458 12082 raft_consensus.cc:357] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.943506 12082 raft_consensus.cc:383] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:57.944052 12082 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:57.945426 12082 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:57.954210 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 75604.2us
W20250113 08:57:57.955559 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 56153.6us
I20250113 08:57:57.899063 12585 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20250113 08:57:57.957412 11856 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:57.960064 11931 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:57.926637 12451 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:57.943866 12470 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:57.953497 12592 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:57:57.965080 12347 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:57.974900 12347 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:57.963096 12582 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:57.975346 12347 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:57.995118 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:57.995864 12036 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:58.002089 12451 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:57.982349 12592 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.025944 12592 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.026594 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:57.984066 12610 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:57:57.983459 12582 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:58.030305 12582 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.029749 12610 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:58.032320 12007 leader_election.cc:343] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:58.032861 12610 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.047473 12082 ts_tablet_manager.cc:1428] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.214s user 0.009s sys 0.000s
I20250113 08:57:57.970556 12470 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.037099 12084 tablet_bootstrap.cc:654] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:58.063341 12556 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:58.065428 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:57:58.072809 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 112573us
I20250113 08:57:58.072091 12082 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
W20250113 08:57:58.071741 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 95183.6us
W20250113 08:57:58.075014 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 110141us
I20250113 08:57:58.071400 12581 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:57:58.064864 12556 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:57:58.100939 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 44830.8us
I20250113 08:57:58.089377 12619 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:58.107580 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:58.119239 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.127110 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.165788 12007 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:57:58.171202 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 55993.1us
I20250113 08:57:58.121567 12461 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:58.173744 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:58.062640 12622 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.076s
I20250113 08:57:58.194289 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.196483 12470 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.176473 12589 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.183929 11888 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:58.227255 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.113231 12619 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.274772 11858 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:58.225075 12610 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:57:58.278201 12597 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.280232 12608 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:58.280712 12608 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.283129 12608 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.272046 12461 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:58.284864 12461 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.286454 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.298516 12347 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.301605 12350 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250113 08:57:58.302724 12629 log.cc:926] Time spent T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20250113 08:57:58.303177 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:58.304013 12035 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 0.
I20250113 08:57:58.306485 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.307353 12556 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.316886 11856 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:57:58.317793 11858 leader_election.cc:343] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:58.301066 12347 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:57:58.326057 11858 consensus_peers.cc:487] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Couldn't send request to peer 37bd4bed3ee6423c8dac0cef3bf6882a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:58.326270 12007 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.331620 12621 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.323330 12598 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:58.339164 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.295177 12610 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.338847 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:57:58.341296 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 52270.6us
I20250113 08:57:58.341851 12610 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.341950 12035 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.324726 12615 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:58.338418 12598 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:58.357203 12600 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:58.368105 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.369524 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:58.370591 12615 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.367488 12598 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.383754 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.402662 12619 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.378408 12600 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.406172 12600 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.411796 12600 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:57:58.389065 12084 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:58.413717 12084 ts_tablet_manager.cc:1397] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.678s user 0.043s sys 0.028s
I20250113 08:57:58.416725 12582 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.417984 12615 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.417814 12600 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.419348 12582 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.421710 11889 raft_consensus.cc:2388] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:58.378400 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:58.428385 12082 tablet_bootstrap.cc:654] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:58.431301 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.438900 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:57:58.440537 12636 log.cc:926] Time spent T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.103s user 0.000s sys 0.002s
I20250113 08:57:58.446704 12006 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:58.447587 12586 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:58.448032 12586 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:58.448426 12586 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.458290 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:58.460954 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:58.460891 12084 raft_consensus.cc:357] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.461444 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.461833 12084 raft_consensus.cc:383] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:58.462129 11960 raft_consensus.cc:2388] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:57:58.462208 12084 raft_consensus.cc:738] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:58.463030 12084 consensus_queue.cc:260] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.464004 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.466095 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.466588 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.421484 12600 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.472971 12007 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.480599 11931 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.480712 12007 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.482645 12007 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.482775 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:58.477902 11856 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:58.488749 12084 ts_tablet_manager.cc:1428] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.075s user 0.008s sys 0.000s
I20250113 08:57:58.489342 12630 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.492956 12608 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.493688 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:58.483722 12590 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.495404 12598 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.483549 12470 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.503834 12084 tablet_bootstrap.cc:492] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:58.505316 11788 catalog_manager.cc:5526] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:58.509341 12461 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.510234 11788 catalog_manager.cc:5526] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } }
I20250113 08:57:58.512826 11932 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.513703 12466 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.524035 12461 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.489470 12619 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:57:58.529141 11856 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:58.525058 12035 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:58.531670 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:58.532245 12597 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250113 08:57:58.534327 11858 leader_election.cc:343] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:58.536490 12598 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.536839 11931 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.541198 12556 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:58.500222 12586 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.542717 12556 raft_consensus.cc:695] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:58.550434 12556 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.547209 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:58.555092 12082 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:58.555541 12082 ts_tablet_manager.cc:1397] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.491s user 0.020s sys 0.009s
W20250113 08:57:58.556229 11858 leader_election.cc:343] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:58.558838 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:58.559677 12042 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.565692 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:58.574807 12037 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:58.575933 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:57:58.576746 12598 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 08:57:58.567831 12084 tablet_bootstrap.cc:654] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:58.585954 12042 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:57:58.587874 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:58.587340 11856 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:58.588685 11891 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:58.589085 12583 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:58.589728 12583 raft_consensus.cc:695] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:57:58.590740 12583 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.601755 12586 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.602355 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:57:58.604696 12007 leader_election.cc:343] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:58.605115 12007 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:58.605968 12460 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 08:57:58.608266 12082 raft_consensus.cc:357] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.608834 12082 raft_consensus.cc:383] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:58.609017 12082 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:58.613441 12082 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.618245 12082 ts_tablet_manager.cc:1428] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.062s user 0.006s sys 0.001s
I20250113 08:57:58.619192 12082 tablet_bootstrap.cc:492] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Bootstrap starting.
I20250113 08:57:58.639226 12084 tablet_bootstrap.cc:492] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:58.639816 12084 ts_tablet_manager.cc:1397] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.136s user 0.011s sys 0.006s
I20250113 08:57:58.641444 12082 tablet_bootstrap.cc:654] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:58.643539 12084 raft_consensus.cc:357] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.644263 12084 raft_consensus.cc:383] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:58.644522 12084 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:58.645264 12084 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.648116 12084 ts_tablet_manager.cc:1428] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250113 08:57:58.649081 12084 tablet_bootstrap.cc:492] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:58.657644 12084 tablet_bootstrap.cc:654] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:58.662657 12082 tablet_bootstrap.cc:492] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: No bootstrap required, opened a new log
I20250113 08:57:58.663240 12082 ts_tablet_manager.cc:1397] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.019s
I20250113 08:57:58.670013 12084 tablet_bootstrap.cc:492] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:58.670483 12084 ts_tablet_manager.cc:1397] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.010s
I20250113 08:57:58.673460 12084 raft_consensus.cc:357] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.674162 12084 raft_consensus.cc:383] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:58.674424 12084 raft_consensus.cc:738] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:58.675007 12084 consensus_queue.cc:260] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.678187 12084 ts_tablet_manager.cc:1428] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250113 08:57:58.683334 12084 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:58.692948 12084 tablet_bootstrap.cc:654] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:58.666373 12082 raft_consensus.cc:357] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.700387 12082 raft_consensus.cc:383] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:58.700687 12082 raft_consensus.cc:738] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Initialized, Role: FOLLOWER
I20250113 08:57:58.701308 12082 consensus_queue.cc:260] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.711441 11960 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.717228 11790 catalog_manager.cc:5526] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:58.703857 12082 ts_tablet_manager.cc:1428] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20250113 08:57:58.724381 11962 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.736359 11788 catalog_manager.cc:5526] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:58.739843 12084 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:58.740422 12084 ts_tablet_manager.cc:1397] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.007s
I20250113 08:57:58.750335 12084 raft_consensus.cc:357] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.763033 12084 raft_consensus.cc:383] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:58.763404 12084 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:58.764143 12084 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.767736 12084 ts_tablet_manager.cc:1428] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s
I20250113 08:57:58.768811 12084 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:58.807721 12084 tablet_bootstrap.cc:654] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:58.820412 12460 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:58.821089 12460 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.824486 12460 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.829887 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.830487 11891 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 0.
I20250113 08:57:58.829667 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:57:58.831665 12006 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:57:58.833051 12637 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:58.833541 12637 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:58.833935 12637 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
W20250113 08:57:58.838440 12007 leader_election.cc:343] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:58.852653 12637 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.855291 12637 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:57:58.855762 12556 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:58.856534 12556 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.864908 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:58.865887 12042 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:58.863370 12556 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:58.868502 11931 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:58.869463 12556 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:57:58.869935 12556 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:57:58.871212 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:57:58.870362 12556 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.878613 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:58.879786 11889 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 0.
I20250113 08:57:58.903501 12556 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.906996 12556 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:57:58.908432 12007 leader_election.cc:343] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:57:58.913292 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:58.922101 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:57:58.930241 11889 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.934088 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:57:58.934780 11891 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.936724 12039 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:58.943783 11889 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:57:58.945073 12006 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:57:58.946036 12637 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:58.946516 12637 raft_consensus.cc:695] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:57:58.947451 12637 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.952454 11891 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:58.954154 12039 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:57:58.955439 11931 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:57:58.956280 12556 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:57:58.957469 12556 raft_consensus.cc:695] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:57:58.975628 12556 consensus_queue.cc:237] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:58.982779 12615 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:58.998482 11790 catalog_manager.cc:5526] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:57:58.997184 12614 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:57:59.011077 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:57:59.002938 12614 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:59.027303 12614 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:59.026717 11788 catalog_manager.cc:5526] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:57:59.036253 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:59.056993 12042 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:59.060180 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:59.061019 11886 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:57:59.063712 12084 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:59.066673 12084 ts_tablet_manager.cc:1397] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.298s user 0.066s sys 0.047s
I20250113 08:57:59.068643 11932 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:59.071430 12464 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:59.080422 12084 raft_consensus.cc:357] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:59.081192 12084 raft_consensus.cc:383] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:59.086916 12084 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
W20250113 08:57:59.074918 12656 log.cc:926] Time spent T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250113 08:57:59.088104 12615 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:59.094293 12084 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:59.096220 12084 ts_tablet_manager.cc:1428] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
I20250113 08:57:59.100347 12464 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:59.100818 12084 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Bootstrap starting.
I20250113 08:57:59.111307 12084 tablet_bootstrap.cc:654] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Neither blocks nor log segments found. Creating new log.
I20250113 08:57:59.124866 12084 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: No bootstrap required, opened a new log
I20250113 08:57:59.125474 12084 ts_tablet_manager.cc:1397] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.002s
I20250113 08:57:59.128182 12084 raft_consensus.cc:357] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:59.129575 12084 raft_consensus.cc:383] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:57:59.129891 12084 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Initialized, Role: FOLLOWER
I20250113 08:57:59.130638 12084 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:59.133251 12084 ts_tablet_manager.cc:1428] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20250113 08:57:59.156455 11962 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:59.182282 12608 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:59.203718 12608 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:59.238554 12464 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:57:59.239073 12464 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:57:59.241731 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:57:59.242516 11887 raft_consensus.cc:2388] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 1: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:57:59.244433 12464 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:57:59.246043 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:57:59.248535 11931 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:57:59.249411 12464 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:57:59.259397 12464 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:57:59.428776 12556 consensus_queue.cc:1035] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:59.441604 12466 consensus_queue.cc:1035] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:57:59.501118 12619 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 08:57:59.513172 12637 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 08:58:00.055795 11839 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250113 08:58:00.425872 12608 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:00.426465 12608 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:00.429427 12608 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:00.430845 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:00.431525 11962 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:00.432637 11858 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:00.434978 12608 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:00.435328 12608 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:00.435652 12608 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:00.434150 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:00.442286 12039 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:00.444908 12608 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:00.456915 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:00.457706 12037 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:00.460081 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:00.455511 12608 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:00.460697 11967 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:00.468109 12037 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:00.469293 11967 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:00.469373 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:00.470286 12608 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:00.470769 12608 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:00.473603 12608 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:00.486923 11788 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 0 to 2, leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:00.832139 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:58:00.858580 12675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:58:00.864511 12677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:58:00.880266 10702 server_base.cc:1034] running on GCE node
W20250113 08:58:00.890265 12674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:58:00.893169 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 08:58:00.893543 10702 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 08:58:00.893744 10702 hybrid_clock.cc:648] HybridClock initialized: now 1736758680893724 us; error 0 us; skew 500 ppm
I20250113 08:58:00.894596 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:58:00.910303 10702 webserver.cc:458] Webserver started at http://127.10.115.132:39937/ using document root <none> and password file <none>
I20250113 08:58:00.911018 10702 fs_manager.cc:362] Metadata directory not provided
I20250113 08:58:00.911310 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:58:00.911675 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:58:00.913419 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-3-root/instance:
uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
format_stamp: "Formatted at 2025-01-13 08:58:00 on dist-test-slave-lrhs"
I20250113 08:58:00.921437 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250113 08:58:00.928310 12683 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:58:00.929463 10702 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250113 08:58:00.929934 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-3-root
uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
format_stamp: "Formatted at 2025-01-13 08:58:00 on dist-test-slave-lrhs"
I20250113 08:58:00.930506 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:58:00.936316 12042 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250113 08:58:00.937810 12678 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:58:00.974606 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:58:00.976747 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:58:00.986141 10702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:58:00.998414 11967 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250113 08:58:00.999768 12678 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:58:01.011413 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:58:01.011958 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20250113 08:58:01.012404 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:58:01.012704 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:58:01.205695 10702 rpc_server.cc:307] RPC server started. Bound to: 127.10.115.132:36511
I20250113 08:58:01.205938 12750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.115.132:36511 every 8 connection(s)
I20250113 08:58:01.220690 10702 auto_leader_rebalancer-test.cc:413] add a tserver: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:01.227357 12751 heartbeater.cc:346] Connected to a master server at 127.10.115.190:41823
I20250113 08:58:01.227885 12751 heartbeater.cc:463] Registering TS with master...
I20250113 08:58:01.228837 12751 heartbeater.cc:510] Master 127.10.115.190:41823 requested a full tablet report, sending...
I20250113 08:58:01.231055 11788 ts_manager.cc:194] Registered new tserver with Master: 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:01.232757 11788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44470
I20250113 08:58:02.231070 11962 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:02.236038 12751 heartbeater.cc:502] Master 127.10.115.190:41823 was elected leader, sending a full tablet report...
I20250113 08:58:02.244520 12035 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:02.246112 11886 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:02.247478 12466 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:02.248642 12556 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250113 08:58:02.273177 11934 consensus_peers.cc:487] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1527f41e7e954abc8622a5f5da292602. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:02.284340 12556 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.291132 11887 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.297806 12035 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.333892 11788 catalog_manager.cc:5526] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.379443 11887 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:02.415215 11962 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:02.417317 12042 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:02.418116 12769 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:02.421559 12608 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:02.450750 12769 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.458909 12042 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.454211 11962 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
W20250113 08:58:02.471403 11859 consensus_peers.cc:487] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18dede64eda24dff8bfe2f979a4115d5. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:02.484596 11790 catalog_manager.cc:5526] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.637383 12042 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:02.675817 11967 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:02.679358 11887 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:02.679442 12619 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:02.693857 12668 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20250113 08:58:02.738682 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 46837.9us
W20250113 08:58:02.760856 12008 consensus_peers.cc:487] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67a8b82ef293439e815f49903f8e5dbb. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:02.767414 12781 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.772498 11967 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.793757 12035 consensus_queue.cc:237] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:02.800449 11789 catalog_manager.cc:5526] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:02.794744 11886 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.833137 11962 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:02.837069 12781 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:02.844502 11891 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:02.846191 12008 consensus_peers.cc:487] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: badac2289c8b4a10915cca20182fc423. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:02.909801 12668 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:02.939769 12785 ts_tablet_manager.cc:927] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:02.955390 12781 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.974179 11967 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.975250 11886 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:02.986119 11789 catalog_manager.cc:5526] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:03.003962 12785 tablet_copy_client.cc:323] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:03.044062 11978 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 1527f41e7e954abc8622a5f5da292602 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:03.044977 11978 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 1527f41e7e954abc8622a5f5da292602 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-1527f41e7e954abc8622a5f5da292602
I20250113 08:58:03.056803 11978 tablet_copy_source_session.cc:215] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:03.060669 12791 ts_tablet_manager.cc:927] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:03.067366 12785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1527f41e7e954abc8622a5f5da292602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:03.088200 12791 tablet_copy_client.cc:323] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:03.104193 12785 tablet_copy_client.cc:806] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:03.105274 12785 tablet_copy_client.cc:670] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:03.112690 12785 tablet_copy_client.cc:538] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:03.122066 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 18dede64eda24dff8bfe2f979a4115d5 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:03.122849 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 18dede64eda24dff8bfe2f979a4115d5 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-18dede64eda24dff8bfe2f979a4115d5
I20250113 08:58:03.127338 12785 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:03.135224 11903 tablet_copy_source_session.cc:215] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:03.139184 12791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dede64eda24dff8bfe2f979a4115d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:03.208228 12785 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:03.209563 12785 tablet_bootstrap.cc:492] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:03.210633 12785 ts_tablet_manager.cc:1397] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.085s user 0.031s sys 0.039s
I20250113 08:58:03.214553 12785 raft_consensus.cc:357] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.216589 12785 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:03.230659 12795 ts_tablet_manager.cc:927] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:03.237607 12791 tablet_copy_client.cc:806] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:03.238363 12791 tablet_copy_client.cc:670] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:03.217445 12785 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.245175 12791 tablet_copy_client.cc:538] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:03.254688 12795 tablet_copy_client.cc:323] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:03.258845 12785 ts_tablet_manager.cc:1428] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.048s user 0.009s sys 0.006s
I20250113 08:58:03.260936 11978 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-1527f41e7e954abc8622a5f5da292602 received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:03.261566 11978 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-1527f41e7e954abc8622a5f5da292602 on tablet 1527f41e7e954abc8622a5f5da292602 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:03.281284 12791 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:03.287120 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 67a8b82ef293439e815f49903f8e5dbb from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:03.287768 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 67a8b82ef293439e815f49903f8e5dbb from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-67a8b82ef293439e815f49903f8e5dbb
I20250113 08:58:03.296663 12052 tablet_copy_source_session.cc:215] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:03.300359 12795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67a8b82ef293439e815f49903f8e5dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:03.302265 12785 ts_tablet_manager.cc:927] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:03.350057 12785 tablet_copy_client.cc:323] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:03.353983 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet badac2289c8b4a10915cca20182fc423 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:03.354540 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet badac2289c8b4a10915cca20182fc423 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-badac2289c8b4a10915cca20182fc423
I20250113 08:58:03.358705 12791 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:03.359611 12791 tablet_bootstrap.cc:492] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:03.360285 12791 ts_tablet_manager.cc:1397] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.079s user 0.017s sys 0.005s
I20250113 08:58:03.362253 12052 tablet_copy_source_session.cc:215] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:03.363265 12791 raft_consensus.cc:357] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.364243 12791 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:03.366645 12785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet badac2289c8b4a10915cca20182fc423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:03.378929 12795 tablet_copy_client.cc:806] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:03.379627 12795 tablet_copy_client.cc:670] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:03.365134 12791 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.385519 12791 ts_tablet_manager.cc:1428] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.025s user 0.008s sys 0.002s
I20250113 08:58:03.387439 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-18dede64eda24dff8bfe2f979a4115d5 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:03.387964 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-18dede64eda24dff8bfe2f979a4115d5 on tablet 18dede64eda24dff8bfe2f979a4115d5 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:03.389022 12785 tablet_copy_client.cc:806] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:03.389658 12785 tablet_copy_client.cc:670] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:03.392485 12795 tablet_copy_client.cc:538] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:03.397811 12785 tablet_copy_client.cc:538] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:03.407466 12795 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:03.410553 12785 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:03.419173 12725 raft_consensus.cc:1212] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:03.460639 12785 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:03.461771 12785 tablet_bootstrap.cc:492] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:03.462514 12785 ts_tablet_manager.cc:1397] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.052s user 0.025s sys 0.007s
I20250113 08:58:03.474347 12785 raft_consensus.cc:357] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.475612 12785 raft_consensus.cc:738] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:03.476377 12785 consensus_queue.cc:260] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.478749 12785 ts_tablet_manager.cc:1428] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250113 08:58:03.480825 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-badac2289c8b4a10915cca20182fc423 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:03.481302 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-badac2289c8b4a10915cca20182fc423 on tablet badac2289c8b4a10915cca20182fc423 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:03.558836 12725 raft_consensus.cc:1212] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250113 08:58:03.570501 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68338.4us
I20250113 08:58:03.585572 12795 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:03.586619 12795 tablet_bootstrap.cc:492] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:03.587327 12795 ts_tablet_manager.cc:1397] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.180s user 0.065s sys 0.068s
I20250113 08:58:03.592936 12795 raft_consensus.cc:357] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.594099 12795 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:03.594893 12795 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:03.597743 12795 ts_tablet_manager.cc:1428] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250113 08:58:03.600911 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-67a8b82ef293439e815f49903f8e5dbb received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:03.601431 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-67a8b82ef293439e815f49903f8e5dbb on tablet 67a8b82ef293439e815f49903f8e5dbb with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:04.542313 12816 raft_consensus.cc:1059] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:04.544997 12816 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:04.578045 12804 raft_consensus.cc:1059] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:04.580806 12804 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:04.629747 12725 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:04.632305 12827 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:04.637287 11885 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:04.652777 12806 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:04.655668 12726 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:04.658591 12831 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:04.660188 12724 raft_consensus.cc:1212] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:04.661234 12726 raft_consensus.cc:1212] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:04.678274 11960 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:04.680539 12813 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:04.688112 12042 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:04.690001 12038 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:04.697537 12806 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:04.701138 12813 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:04.718478 12831 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.727181 12806 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.729907 12035 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.743356 11885 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.759133 12042 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.759332 11960 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.759470 11790 catalog_manager.cc:5526] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:04.764019 12724 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.723112 12726 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.758152 11789 catalog_manager.cc:5526] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:04.814319 11960 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:04.820597 12726 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:04.820930 11885 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:04.822188 12827 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:04.825337 11885 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:04.824780 12827 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:04.831887 12724 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:04.833972 12831 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:04.839437 12816 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.843173 12042 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:04.843207 11890 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.848608 12724 raft_consensus.cc:2949] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.849452 12831 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:04.861109 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1527f41e7e954abc8622a5f5da292602 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:04.861922 12818 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.871240 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 18dede64eda24dff8bfe2f979a4115d5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:04.875196 12042 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.880136 12726 raft_consensus.cc:2949] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:04.902228 11788 catalog_manager.cc:5526] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:04.911047 11777 catalog_manager.cc:5049] ChangeConfig:REMOVE_PEER RPC for tablet 1527f41e7e954abc8622a5f5da292602 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20250113 08:58:04.938383 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 18dede64eda24dff8bfe2f979a4115d5 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:04.956228 12845 tablet_replica.cc:331] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:04.957494 12845 raft_consensus.cc:2238] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:04.958269 12845 raft_consensus.cc:2267] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:04.964252 11789 catalog_manager.cc:5526] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:04.969657 12845 ts_tablet_manager.cc:1905] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:04.980618 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1527f41e7e954abc8622a5f5da292602 with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:04.986677 12845 ts_tablet_manager.cc:1918] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:04.987447 12845 log.cc:1198] T 18dede64eda24dff8bfe2f979a4115d5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/18dede64eda24dff8bfe2f979a4115d5
I20250113 08:58:04.990999 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 18dede64eda24dff8bfe2f979a4115d5 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:04.996352 12846 tablet_replica.cc:331] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:04.998656 12846 raft_consensus.cc:2238] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:04.999467 12846 raft_consensus.cc:2267] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:05.010279 12846 ts_tablet_manager.cc:1905] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:05.028201 12846 ts_tablet_manager.cc:1918] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:05.028661 12846 log.cc:1198] T 1527f41e7e954abc8622a5f5da292602 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/1527f41e7e954abc8622a5f5da292602
I20250113 08:58:05.036473 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 1527f41e7e954abc8622a5f5da292602 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:05.065099 12809 raft_consensus.cc:1059] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:05.067615 12809 consensus_queue.cc:237] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:05.077195 11965 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:05.079481 12726 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:05.079929 12820 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:05.081496 12808 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:05.086169 11890 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:05.087651 12814 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:05.151005 12809 raft_consensus.cc:1059] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:05.154232 12814 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.158571 12809 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:05.157776 11890 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.166581 11963 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.200453 11791 catalog_manager.cc:5526] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.205710 12726 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:05.205961 11963 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:05.207525 12808 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:05.207892 11890 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:05.213035 12820 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:05.220227 12820 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:05.221845 12724 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.236855 12726 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.232537 12808 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.245939 12037 consensus_queue.cc:237] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:05.270625 11965 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.272625 12724 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:05.279320 12814 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:05.280190 11963 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:05.283329 12814 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:05.284260 11791 catalog_manager.cc:5526] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.289741 12810 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.292085 11963 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.306092 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet badac2289c8b4a10915cca20182fc423 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:05.325292 11890 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.332198 11789 catalog_manager.cc:5526] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.327929 12726 raft_consensus.cc:2949] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.365480 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet badac2289c8b4a10915cca20182fc423 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:05.367797 12855 tablet_replica.cc:331] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:05.369014 12855 raft_consensus.cc:2238] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:05.369868 12855 raft_consensus.cc:2267] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:05.377067 12855 ts_tablet_manager.cc:1905] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:05.387676 12037 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:05.396342 12726 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:05.398002 12810 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:05.398828 11963 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:05.400564 12809 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:05.409569 11963 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.406159 12809 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.409564 12726 raft_consensus.cc:2949] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.435372 12855 ts_tablet_manager.cc:1918] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:05.436250 12855 log.cc:1198] T badac2289c8b4a10915cca20182fc423 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/badac2289c8b4a10915cca20182fc423
I20250113 08:58:05.441412 11791 catalog_manager.cc:5526] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:05.443993 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet badac2289c8b4a10915cca20182fc423 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
W20250113 08:58:05.461711 12063 debug-util.cc:398] Leaking SignalData structure 0x7b08001d62a0 after lost signal to thread 10705
W20250113 08:58:05.465550 12063 debug-util.cc:398] Leaking SignalData structure 0x7b08001073a0 after lost signal to thread 11825
I20250113 08:58:05.467943 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a8b82ef293439e815f49903f8e5dbb with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:05.453081 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 67a8b82ef293439e815f49903f8e5dbb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:05.477640 12855 tablet_replica.cc:331] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
W20250113 08:58:05.466898 12063 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9ac0 after lost signal to thread 11917
W20250113 08:58:05.482708 12063 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9ae0 after lost signal to thread 11992
W20250113 08:58:05.483933 12063 debug-util.cc:398] Leaking SignalData structure 0x7b08001b1d60 after lost signal to thread 12066
W20250113 08:58:05.485209 12063 debug-util.cc:398] Leaking SignalData structure 0x7b08001995e0 after lost signal to thread 12750
I20250113 08:58:05.525071 12855 raft_consensus.cc:2238] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:05.525897 12855 raft_consensus.cc:2267] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:05.529084 12855 ts_tablet_manager.cc:1905] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:05.588438 12855 ts_tablet_manager.cc:1918] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:05.588986 12855 log.cc:1198] T 67a8b82ef293439e815f49903f8e5dbb P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/67a8b82ef293439e815f49903f8e5dbb
I20250113 08:58:05.590598 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 67a8b82ef293439e815f49903f8e5dbb (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:06.618552 11885 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:06.627781 11859 consensus_peers.cc:487] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 218ea9985e1d40a48892a8e01aaaef39. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:06.628093 12042 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.629683 12823 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:06.630579 11963 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.632608 12823 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:06.653052 12825 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.655191 12042 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.664590 11963 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.685084 11791 catalog_manager.cc:5526] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.707767 11885 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:06.721611 11859 consensus_peers.cc:487] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de29a6701ddb4906a7eff4d34bf97ab5. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:06.721887 11963 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.721154 12038 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.723376 12825 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:06.724498 12863 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:06.746735 12863 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.749080 12042 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.759011 11963 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.765902 12038 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:06.784660 11790 catalog_manager.cc:5526] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
W20250113 08:58:06.791185 12008 consensus_peers.cc:487] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 364888f946914390a5a07be6735a4cc9. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:06.794171 11963 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.794896 11885 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.795846 12814 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:06.797221 12856 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:06.823312 12820 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.829838 11963 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.831827 11885 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.858563 11965 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:06.872591 11790 catalog_manager.cc:5526] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
W20250113 08:58:06.890758 11934 consensus_peers.cc:487] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87a2e0ccdce34284b13fa0924326b262. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:06.891636 12038 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.893007 11885 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:06.893532 12827 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:06.894965 12822 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:06.933771 12038 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.932181 12827 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.956360 11885 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:06.964464 11788 catalog_manager.cc:5526] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:07.050083 12880 ts_tablet_manager.cc:927] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:07.059417 12880 tablet_copy_client.cc:323] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:07.061590 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 218ea9985e1d40a48892a8e01aaaef39 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:07.062309 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 218ea9985e1d40a48892a8e01aaaef39 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-218ea9985e1d40a48892a8e01aaaef39
I20250113 08:58:07.078352 11903 tablet_copy_source_session.cc:215] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:07.084368 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218ea9985e1d40a48892a8e01aaaef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:07.162925 12880 tablet_copy_client.cc:806] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:07.163817 12880 tablet_copy_client.cc:670] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:07.171675 12880 tablet_copy_client.cc:538] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:07.186218 12880 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:07.227100 12882 ts_tablet_manager.cc:927] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:07.232477 12882 tablet_copy_client.cc:323] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:07.234069 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 364888f946914390a5a07be6735a4cc9 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:07.234601 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 364888f946914390a5a07be6735a4cc9 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-364888f946914390a5a07be6735a4cc9
I20250113 08:58:07.241930 12052 tablet_copy_source_session.cc:215] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:07.245151 12882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364888f946914390a5a07be6735a4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:07.234019 12884 ts_tablet_manager.cc:927] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:07.254838 12880 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:07.255813 12880 tablet_bootstrap.cc:492] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:07.256786 12880 ts_tablet_manager.cc:1397] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.071s user 0.030s sys 0.003s
I20250113 08:58:07.267606 12884 tablet_copy_client.cc:323] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:07.271386 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet de29a6701ddb4906a7eff4d34bf97ab5 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:07.272107 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet de29a6701ddb4906a7eff4d34bf97ab5 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-de29a6701ddb4906a7eff4d34bf97ab5
I20250113 08:58:07.280156 12882 tablet_copy_client.cc:806] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:07.280925 12882 tablet_copy_client.cc:670] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:07.286687 12882 tablet_copy_client.cc:538] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:07.289389 11903 tablet_copy_source_session.cc:215] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:07.266734 12880 raft_consensus.cc:357] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.292907 12880 raft_consensus.cc:738] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:07.293965 12880 consensus_queue.cc:260] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.297832 12882 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:07.319751 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de29a6701ddb4906a7eff4d34bf97ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:07.332341 12880 ts_tablet_manager.cc:1428] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.075s user 0.006s sys 0.015s
I20250113 08:58:07.336477 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-218ea9985e1d40a48892a8e01aaaef39 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:07.337152 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-218ea9985e1d40a48892a8e01aaaef39 on tablet 218ea9985e1d40a48892a8e01aaaef39 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:07.351397 12884 tablet_copy_client.cc:806] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:07.352097 12884 tablet_copy_client.cc:670] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:07.357477 12884 tablet_copy_client.cc:538] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:07.369565 12882 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:07.370741 12882 tablet_bootstrap.cc:492] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:07.370708 12884 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:07.371819 12882 ts_tablet_manager.cc:1397] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.074s user 0.022s sys 0.023s
I20250113 08:58:07.376147 12882 raft_consensus.cc:357] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.389727 12882 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:07.390581 12882 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.398346 12882 ts_tablet_manager.cc:1428] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.026s user 0.008s sys 0.002s
I20250113 08:58:07.401104 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-364888f946914390a5a07be6735a4cc9 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:07.401746 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-364888f946914390a5a07be6735a4cc9 on tablet 364888f946914390a5a07be6735a4cc9 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:07.404099 12880 ts_tablet_manager.cc:927] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:07.435422 12880 tablet_copy_client.cc:323] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:07.437500 12884 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:07.437530 11978 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 87a2e0ccdce34284b13fa0924326b262 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:07.438539 11978 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 87a2e0ccdce34284b13fa0924326b262 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-87a2e0ccdce34284b13fa0924326b262
I20250113 08:58:07.438779 12884 tablet_bootstrap.cc:492] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:07.439633 12884 ts_tablet_manager.cc:1397] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.069s user 0.027s sys 0.006s
I20250113 08:58:07.443444 12884 raft_consensus.cc:357] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.444571 12884 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:07.445482 12884 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.448925 12884 ts_tablet_manager.cc:1428] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.009s user 0.010s sys 0.001s
I20250113 08:58:07.451375 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-de29a6701ddb4906a7eff4d34bf97ab5 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:07.451870 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-de29a6701ddb4906a7eff4d34bf97ab5 on tablet de29a6701ddb4906a7eff4d34bf97ab5 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:07.466104 11978 tablet_copy_source_session.cc:215] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:07.470140 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a2e0ccdce34284b13fa0924326b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:07.490960 12726 raft_consensus.cc:1212] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:07.542765 12880 tablet_copy_client.cc:806] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:07.543583 12880 tablet_copy_client.cc:670] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
W20250113 08:58:07.556309 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45747.5us
I20250113 08:58:07.563413 12880 tablet_copy_client.cc:538] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:07.576260 12880 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:07.613898 12880 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:07.615025 12880 tablet_bootstrap.cc:492] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:07.615849 12880 ts_tablet_manager.cc:1397] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.009s
I20250113 08:58:07.619930 12880 raft_consensus.cc:357] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.621026 12880 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:07.621968 12880 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:07.625574 12880 ts_tablet_manager.cc:1428] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.009s user 0.006s sys 0.006s
I20250113 08:58:07.629737 11978 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-87a2e0ccdce34284b13fa0924326b262 received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:07.630244 11978 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-87a2e0ccdce34284b13fa0924326b262 on tablet 87a2e0ccdce34284b13fa0924326b262 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:07.641134 12726 raft_consensus.cc:1212] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:07.740511 12724 raft_consensus.cc:1212] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:07.953799 12892 raft_consensus.cc:1059] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:07.956300 12892 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:07.959144 12726 raft_consensus.cc:1212] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:07.972076 12726 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:07.973743 12893 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:07.985988 12041 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:07.989620 11965 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:07.989409 12825 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:07.992889 12825 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250113 08:58:08.021747 12897 log.cc:926] Time spent T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250113 08:58:08.027767 12893 raft_consensus.cc:1059] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:08.030565 12893 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:08.055625 12891 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.058456 12726 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.060178 12892 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:08.072811 11963 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.080073 11965 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.084010 12892 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:08.094337 12724 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.105818 12037 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.111497 11788 catalog_manager.cc:5526] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:08.122179 12904 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:08.150854 12820 raft_consensus.cc:1059] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:08.165808 12903 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.167899 12038 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.175536 12726 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.154168 12820 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:08.219169 12038 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.236544 11965 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.239087 12724 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.241525 11965 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.240342 12909 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20250113 08:58:08.296388 12874 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:08.326763 11893 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.339425 11893 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:08.359812 12920 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:08.365267 11960 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.371253 12891 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:08.365150 12856 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.378011 12912 raft_consensus.cc:1059] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:08.380106 12724 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.354460 11790 catalog_manager.cc:5526] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:08.382256 12903 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.382649 12912 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:08.381582 12726 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.387831 12893 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:08.399502 11960 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.401970 12723 raft_consensus.cc:2949] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.423727 11963 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.434613 11885 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.438022 11790 catalog_manager.cc:5526] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:08.425616 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 218ea9985e1d40a48892a8e01aaaef39 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:08.415825 11893 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:08.455241 12039 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:08.483685 12039 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.484580 12722 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.485340 12891 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:08.462705 11790 catalog_manager.cc:5526] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:08.494095 12903 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250113 08:58:08.505538 12892 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.509810 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 218ea9985e1d40a48892a8e01aaaef39 with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:08.515409 12039 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.528010 12722 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.533812 12038 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.542289 11790 catalog_manager.cc:5526] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.544667 12725 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.545831 12827 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:08.550072 11960 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.552145 12879 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250113 08:58:08.554946 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet de29a6701ddb4906a7eff4d34bf97ab5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:08.570658 12856 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250113 08:58:08.578878 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet de29a6701ddb4906a7eff4d34bf97ab5 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:08.580817 11893 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:08.588743 12923 tablet_replica.cc:331] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:08.590523 12923 raft_consensus.cc:2238] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:08.593124 12923 raft_consensus.cc:2267] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:08.579196 12856 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250113 08:58:08.608567 12909 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.615092 12723 raft_consensus.cc:2949] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.623828 12923 ts_tablet_manager.cc:1905] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:08.624329 11960 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.603557 12912 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20250113 08:58:08.657336 11790 catalog_manager.cc:5526] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.665159 12924 tablet_replica.cc:331] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:08.669739 12726 raft_consensus.cc:2949] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.678548 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 364888f946914390a5a07be6735a4cc9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:08.678937 12924 raft_consensus.cc:2238] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:08.679796 12924 raft_consensus.cc:2267] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:08.684008 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 364888f946914390a5a07be6735a4cc9 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:08.680377 12879 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.688014 11893 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.688458 12724 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.698812 12924 ts_tablet_manager.cc:1905] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:08.756574 12042 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.771142 12923 ts_tablet_manager.cc:1918] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:08.771699 12923 log.cc:1198] T 218ea9985e1d40a48892a8e01aaaef39 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/218ea9985e1d40a48892a8e01aaaef39
I20250113 08:58:08.774638 11788 catalog_manager.cc:5526] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:08.778126 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 218ea9985e1d40a48892a8e01aaaef39 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:08.794838 12928 tablet_replica.cc:331] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:08.805416 11963 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:08.823112 12928 raft_consensus.cc:2238] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:08.823935 12928 raft_consensus.cc:2267] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:08.828183 12928 ts_tablet_manager.cc:1905] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:08.840010 11893 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.843050 12912 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:08.844543 12726 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:08.847239 12912 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:08.860778 12879 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.861497 11893 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.880718 12924 ts_tablet_manager.cc:1918] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:08.881655 12924 log.cc:1198] T de29a6701ddb4906a7eff4d34bf97ab5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/de29a6701ddb4906a7eff4d34bf97ab5
I20250113 08:58:08.882052 12724 raft_consensus.cc:2949] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.876412 12928 ts_tablet_manager.cc:1918] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:08.888938 12928 log.cc:1198] T 364888f946914390a5a07be6735a4cc9 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/364888f946914390a5a07be6735a4cc9
I20250113 08:58:08.894323 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 364888f946914390a5a07be6735a4cc9 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:08.908464 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 87a2e0ccdce34284b13fa0924326b262 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:08.912628 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet de29a6701ddb4906a7eff4d34bf97ab5 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:08.922161 11789 catalog_manager.cc:5526] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:08.939108 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a2e0ccdce34284b13fa0924326b262 with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:08.940302 12923 tablet_replica.cc:331] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:08.941671 12923 raft_consensus.cc:2238] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:08.942370 12923 raft_consensus.cc:2267] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:08.945297 12923 ts_tablet_manager.cc:1905] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:08.961105 12923 ts_tablet_manager.cc:1918] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:08.961573 12923 log.cc:1198] T 87a2e0ccdce34284b13fa0924326b262 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/87a2e0ccdce34284b13fa0924326b262
I20250113 08:58:08.962975 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 87a2e0ccdce34284b13fa0924326b262 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:10.018807 12039 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.029222 11965 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:10.029407 12008 consensus_peers.cc:487] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a23cb6607d3a463293b397049ef2d690. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:10.030611 12930 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:10.030911 11893 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:10.032228 12914 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:10.040741 12914 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.042732 11893 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.045866 11965 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.060106 11960 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.061874 11788 catalog_manager.cc:5526] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:10.086541 12042 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:10.086805 11934 consensus_peers.cc:487] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07a14a3957e44925ae0925b645d220eb. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:10.089171 12827 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:10.092830 11889 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:10.094508 12827 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:10.107789 12827 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.110947 12039 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.142681 11889 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.180713 11893 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.218972 12042 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:10.219092 11859 consensus_peers.cc:487] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69aa379773c047589ae2362850997a47. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:10.220374 11960 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:10.221068 12932 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:10.230156 12933 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:10.226111 11789 catalog_manager.cc:5526] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:10.243921 12932 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.260339 11965 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.261438 12042 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.264631 12039 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.283020 11788 catalog_manager.cc:5526] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250113 08:58:10.284142 12008 consensus_peers.cc:487] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dda9aa8769e4146a4dabeadd588c734. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:10.286535 11965 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:10.287876 11889 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:10.288061 12930 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:10.290690 12856 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:10.315141 12856 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.316704 11893 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.328512 11963 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.338423 11788 catalog_manager.cc:5526] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:10.439777 12953 ts_tablet_manager.cc:927] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:10.441989 12953 tablet_copy_client.cc:323] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:10.444782 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet a23cb6607d3a463293b397049ef2d690 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:10.445402 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet a23cb6607d3a463293b397049ef2d690 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-a23cb6607d3a463293b397049ef2d690
I20250113 08:58:10.452349 12052 tablet_copy_source_session.cc:215] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:10.455502 12953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a23cb6607d3a463293b397049ef2d690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:10.471179 12953 tablet_copy_client.cc:806] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:10.471740 12953 tablet_copy_client.cc:670] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:10.475168 12953 tablet_copy_client.cc:538] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:10.483139 12953 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:10.503682 12953 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:10.504604 12953 tablet_bootstrap.cc:492] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:10.505367 12953 ts_tablet_manager.cc:1397] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.022s user 0.023s sys 0.000s
I20250113 08:58:10.508185 12953 raft_consensus.cc:357] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.509088 12953 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:10.509781 12953 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.515506 12953 ts_tablet_manager.cc:1428] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20250113 08:58:10.517465 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a23cb6607d3a463293b397049ef2d690 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:10.517948 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a23cb6607d3a463293b397049ef2d690 on tablet a23cb6607d3a463293b397049ef2d690 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:10.611543 12953 ts_tablet_manager.cc:927] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:10.613046 12953 tablet_copy_client.cc:323] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:10.614387 11978 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 07a14a3957e44925ae0925b645d220eb from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:10.614823 11978 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 07a14a3957e44925ae0925b645d220eb from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-07a14a3957e44925ae0925b645d220eb
I20250113 08:58:10.624284 11978 tablet_copy_source_session.cc:215] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:10.627214 12953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a14a3957e44925ae0925b645d220eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:10.642418 12953 tablet_copy_client.cc:806] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:10.642999 12953 tablet_copy_client.cc:670] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:10.646484 12953 tablet_copy_client.cc:538] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:10.659262 12953 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:10.679299 12953 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:10.680159 12953 tablet_bootstrap.cc:492] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:10.680859 12953 ts_tablet_manager.cc:1397] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20250113 08:58:10.683524 12953 raft_consensus.cc:357] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.684159 12953 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:10.684803 12953 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.690706 12953 ts_tablet_manager.cc:1428] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20250113 08:58:10.694469 11978 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-07a14a3957e44925ae0925b645d220eb received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:10.694943 11978 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-07a14a3957e44925ae0925b645d220eb on tablet 07a14a3957e44925ae0925b645d220eb with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:10.749703 12953 ts_tablet_manager.cc:927] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:10.751291 12953 tablet_copy_client.cc:323] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:10.752599 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 7dda9aa8769e4146a4dabeadd588c734 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:10.753021 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 7dda9aa8769e4146a4dabeadd588c734 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-7dda9aa8769e4146a4dabeadd588c734
I20250113 08:58:10.759830 12052 tablet_copy_source_session.cc:215] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:10.764766 12953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dda9aa8769e4146a4dabeadd588c734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:10.781566 12953 tablet_copy_client.cc:806] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:10.782131 12953 tablet_copy_client.cc:670] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:10.785303 12953 tablet_copy_client.cc:538] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:10.792872 12953 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:10.810990 12953 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:10.811785 12953 tablet_bootstrap.cc:492] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:10.812438 12953 ts_tablet_manager.cc:1397] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20250113 08:58:10.814950 12953 raft_consensus.cc:357] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.817580 12953 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:10.819384 12961 ts_tablet_manager.cc:927] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:10.819034 12953 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.822943 12961 tablet_copy_client.cc:323] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:10.823788 12953 ts_tablet_manager.cc:1428] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20250113 08:58:10.824477 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 69aa379773c047589ae2362850997a47 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:10.824966 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 69aa379773c047589ae2362850997a47 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-69aa379773c047589ae2362850997a47
I20250113 08:58:10.826395 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-7dda9aa8769e4146a4dabeadd588c734 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:10.826761 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-7dda9aa8769e4146a4dabeadd588c734 on tablet 7dda9aa8769e4146a4dabeadd588c734 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:10.832741 11903 tablet_copy_source_session.cc:215] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:10.835875 12961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69aa379773c047589ae2362850997a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:10.850250 12961 tablet_copy_client.cc:806] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:10.850814 12961 tablet_copy_client.cc:670] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:10.854231 12961 tablet_copy_client.cc:538] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:10.863642 12961 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:10.882836 12961 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:10.883764 12961 tablet_bootstrap.cc:492] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:10.884469 12961 ts_tablet_manager.cc:1397] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20250113 08:58:10.887152 12961 raft_consensus.cc:357] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.888105 12961 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:10.888576 12961 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:10.891098 12961 ts_tablet_manager.cc:1428] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250113 08:58:10.893040 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-69aa379773c047589ae2362850997a47 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:10.893509 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-69aa379773c047589ae2362850997a47 on tablet 69aa379773c047589ae2362850997a47 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:11.064168 12724 raft_consensus.cc:1212] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:11.163897 12724 raft_consensus.cc:1212] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:11.326630 12724 raft_consensus.cc:1212] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:11.408656 12724 raft_consensus.cc:1212] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:11.578406 12964 raft_consensus.cc:1059] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:11.580742 12964 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:11.586962 12724 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:11.588333 12944 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:11.591737 11893 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:11.591642 12038 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:11.593223 12827 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:11.598798 12964 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:11.601727 12827 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.608698 11889 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.611218 12042 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.614428 12724 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.624320 12856 raft_consensus.cc:1059] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:11.629556 11789 catalog_manager.cc:5526] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.634753 12856 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:11.654464 12724 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:11.655848 12914 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:11.656553 11963 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:11.659091 11889 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:11.662254 12914 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:11.663277 11963 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:11.671842 12914 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:11.692679 12726 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:11.694056 11893 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:11.694288 12944 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:11.696671 12944 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:11.704500 12723 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.705538 12827 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.721079 12952 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.721144 11889 raft_consensus.cc:2949] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.722750 12724 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.735687 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 07a14a3957e44925ae0925b645d220eb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:11.735939 11965 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.752350 11791 catalog_manager.cc:5526] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.770120 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a14a3957e44925ae0925b645d220eb with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:11.776679 11893 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.798635 11789 catalog_manager.cc:5526] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:11.816905 12979 tablet_replica.cc:331] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:11.818046 12979 raft_consensus.cc:2238] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:11.818852 12979 raft_consensus.cc:2267] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 08:58:11.822383 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 85101.2us
I20250113 08:58:11.822618 12979 ts_tablet_manager.cc:1905] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:11.846580 12039 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:11.855824 12724 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:11.857589 12952 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:11.870375 12952 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:12.037503 12914 raft_consensus.cc:1059] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:12.040697 12724 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:12.136294 12914 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:12.397867 12979 ts_tablet_manager.cc:1918] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:12.399078 12979 log.cc:1198] T 07a14a3957e44925ae0925b645d220eb P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/07a14a3957e44925ae0925b645d220eb
I20250113 08:58:12.504670 11889 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250113 08:58:12.511814 12007 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.002s sys 0.008s
W20250113 08:58:12.512224 12007 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.003s sys 0.008s
W20250113 08:58:12.512668 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 512900us
I20250113 08:58:12.522840 11889 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:12.525998 12982 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:12.532368 12914 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:12.542763 11892 raft_consensus.cc:2949] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:12.628218 11968 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250113 08:58:12.635257 12005 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.012s sys 0.022s
W20250113 08:58:12.635648 12005 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.766s user 0.012s sys 0.022s
I20250113 08:58:12.650224 11790 catalog_manager.cc:5526] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:12.655479 12932 raft_consensus.cc:1059] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:12.658149 12932 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:12.677888 12725 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:12.680341 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a23cb6607d3a463293b397049ef2d690 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:12.681070 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 07a14a3957e44925ae0925b645d220eb (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:12.694355 12914 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:12.749186 12981 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:12.762305 12983 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:12.764382 12725 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:12.816568 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 83949.9us
W20250113 08:58:12.835053 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 99296us
W20250113 08:58:12.837389 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 125541us
I20250113 08:58:12.849704 12725 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:12.874670 11963 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:12.885661 12987 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:12.889886 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet a23cb6607d3a463293b397049ef2d690 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
W20250113 08:58:12.894933 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 48473.3us
I20250113 08:58:12.908478 12041 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:12.937834 12986 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:12.924813 11787 catalog_manager.cc:5526] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:12.969995 13011 tablet_replica.cc:331] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
W20250113 08:58:12.970608 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 78029.1us
W20250113 08:58:12.977504 13010 log.cc:926] Time spent T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.060s user 0.000s sys 0.002s
I20250113 08:58:13.000574 11967 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.034772 13007 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:13.057236 11890 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.063110 13011 raft_consensus.cc:2238] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:13.064393 13011 raft_consensus.cc:2267] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:13.068733 13011 ts_tablet_manager.cc:1905] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:13.127965 13000 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.144618 12725 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.146816 12041 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:13.150918 12034 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.153076 11788 catalog_manager.cc:5526] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:13.155357 12724 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:13.158083 11965 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:13.162575 12856 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:13.165768 13011 ts_tablet_manager.cc:1918] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:13.166522 13011 log.cc:1198] T a23cb6607d3a463293b397049ef2d690 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/a23cb6607d3a463293b397049ef2d690
I20250113 08:58:13.169020 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet a23cb6607d3a463293b397049ef2d690 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:13.177891 12952 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:13.191232 11967 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.187189 12952 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.208001 11965 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.209218 11891 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:13.203385 12725 raft_consensus.cc:2949] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.215781 12724 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:13.216778 12039 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:13.217393 13000 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:13.218382 13006 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:13.226537 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7dda9aa8769e4146a4dabeadd588c734 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:13.231627 12993 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.234862 12724 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.238946 12041 raft_consensus.cc:2949] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.242296 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 69aa379773c047589ae2362850997a47 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:13.242288 11787 catalog_manager.cc:5526] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.258993 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dda9aa8769e4146a4dabeadd588c734 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:13.262679 11788 catalog_manager.cc:5526] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:13.280247 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aa379773c047589ae2362850997a47 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:13.281270 13011 tablet_replica.cc:331] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:13.282389 13011 raft_consensus.cc:2238] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:13.276664 13017 tablet_replica.cc:331] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:13.283000 13011 raft_consensus.cc:2267] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:13.286432 13011 ts_tablet_manager.cc:1905] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:13.288100 13017 raft_consensus.cc:2238] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:13.288905 13017 raft_consensus.cc:2267] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:13.294167 13017 ts_tablet_manager.cc:1905] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:13.303154 13011 ts_tablet_manager.cc:1918] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:13.303637 13011 log.cc:1198] T 69aa379773c047589ae2362850997a47 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/69aa379773c047589ae2362850997a47
I20250113 08:58:13.305320 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 69aa379773c047589ae2362850997a47 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:13.308482 13017 ts_tablet_manager.cc:1918] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:13.308979 13017 log.cc:1198] T 7dda9aa8769e4146a4dabeadd588c734 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/7dda9aa8769e4146a4dabeadd588c734
I20250113 08:58:13.310639 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 7dda9aa8769e4146a4dabeadd588c734 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
W20250113 08:58:13.564770 12747 debug-util.cc:398] Leaking SignalData structure 0x7b0800289fa0 after lost signal to thread 10705
W20250113 08:58:13.566157 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08001a0500 after lost signal to thread 11825
W20250113 08:58:13.567673 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08001f9f60 after lost signal to thread 11917
W20250113 08:58:13.569092 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad800 after lost signal to thread 11992
W20250113 08:58:13.570135 12747 debug-util.cc:398] Leaking SignalData structure 0x7b080013ee60 after lost signal to thread 12066
W20250113 08:58:13.571152 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff540 after lost signal to thread 12750
I20250113 08:58:14.352244 12034 consensus_queue.cc:237] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:14.359669 11967 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:14.360239 12008 consensus_peers.cc:487] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcc31099ea8040b7bfbfe7d779148ca5. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:14.361115 12856 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:14.361595 11891 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:14.362828 13015 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:14.374869 12856 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.380107 11891 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.380120 11965 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.411280 11789 catalog_manager.cc:5526] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.426993 11890 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:14.456614 11859 consensus_peers.cc:487] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 209260c370164f719208bd1404eef071. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:14.480441 12034 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:14.485311 11965 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:14.486285 13026 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:14.493347 13006 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:14.509163 11967 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.507540 12986 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.529198 12041 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.572422 11890 consensus_queue.cc:237] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:14.588171 11788 catalog_manager.cc:5526] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.611878 11967 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:14.614005 13034 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:14.618180 12041 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:14.622154 13036 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250113 08:58:14.625206 11859 consensus_peers.cc:487] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70a6087e1ad14cca8bb3f6d3e0bb1858. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:14.633173 13036 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.634758 11965 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.645304 12039 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.668216 12034 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:14.670003 11787 catalog_manager.cc:5526] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
W20250113 08:58:14.681994 12008 consensus_peers.cc:487] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 705a99b7862446be849d4c843d8084ee. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:14.682196 11890 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:14.684852 13015 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:14.686378 11965 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:14.691290 13025 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:14.705565 13015 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.706616 11891 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.724861 11965 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:14.740826 11787 catalog_manager.cc:5526] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:14.895188 13044 ts_tablet_manager.cc:927] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:14.917997 13046 ts_tablet_manager.cc:927] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:14.920549 13044 tablet_copy_client.cc:323] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:14.926316 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 209260c370164f719208bd1404eef071 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:14.926964 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 209260c370164f719208bd1404eef071 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-209260c370164f719208bd1404eef071
I20250113 08:58:14.927376 13046 tablet_copy_client.cc:323] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:14.930358 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet fcc31099ea8040b7bfbfe7d779148ca5 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:14.930893 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet fcc31099ea8040b7bfbfe7d779148ca5 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-fcc31099ea8040b7bfbfe7d779148ca5
I20250113 08:58:14.941488 11903 tablet_copy_source_session.cc:215] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:14.942946 12052 tablet_copy_source_session.cc:215] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:14.945772 13044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209260c370164f719208bd1404eef071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:14.946539 13046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc31099ea8040b7bfbfe7d779148ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:14.964030 13044 tablet_copy_client.cc:806] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:14.964769 13044 tablet_copy_client.cc:670] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:14.971562 13044 tablet_copy_client.cc:538] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:14.975409 13046 tablet_copy_client.cc:806] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:14.976130 13046 tablet_copy_client.cc:670] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:14.983788 13046 tablet_copy_client.cc:538] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:14.994398 13044 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:15.006950 13046 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:15.040782 13044 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:15.042014 13044 tablet_bootstrap.cc:492] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:15.045338 13044 ts_tablet_manager.cc:1397] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.049s user 0.029s sys 0.002s
I20250113 08:58:15.046746 13046 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:15.047859 13046 tablet_bootstrap.cc:492] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:15.048590 13046 ts_tablet_manager.cc:1397] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.042s user 0.020s sys 0.016s
I20250113 08:58:15.048705 13044 raft_consensus.cc:357] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.049913 13044 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:15.050699 13044 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.051610 13046 raft_consensus.cc:357] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.052624 13046 raft_consensus.cc:738] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:15.053367 13046 consensus_queue.cc:260] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.067009 13044 ts_tablet_manager.cc:1428] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.021s user 0.009s sys 0.005s
I20250113 08:58:15.068871 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-209260c370164f719208bd1404eef071 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:15.069366 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-209260c370164f719208bd1404eef071 on tablet 209260c370164f719208bd1404eef071 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:15.077275 13046 ts_tablet_manager.cc:1428] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.028s user 0.013s sys 0.008s
I20250113 08:58:15.079432 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-fcc31099ea8040b7bfbfe7d779148ca5 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:15.079912 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-fcc31099ea8040b7bfbfe7d779148ca5 on tablet fcc31099ea8040b7bfbfe7d779148ca5 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:15.116981 13046 ts_tablet_manager.cc:927] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:15.122205 13046 tablet_copy_client.cc:323] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:15.123698 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 705a99b7862446be849d4c843d8084ee from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:15.124269 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 705a99b7862446be849d4c843d8084ee from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-705a99b7862446be849d4c843d8084ee
I20250113 08:58:15.138810 12052 tablet_copy_source_session.cc:215] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:15.145334 13046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705a99b7862446be849d4c843d8084ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:15.145641 13044 ts_tablet_manager.cc:927] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:15.149820 13044 tablet_copy_client.cc:323] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:15.153110 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:15.153647 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-70a6087e1ad14cca8bb3f6d3e0bb1858
I20250113 08:58:15.173915 11903 tablet_copy_source_session.cc:215] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:15.181013 13044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:15.195389 13046 tablet_copy_client.cc:806] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:15.196094 13046 tablet_copy_client.cc:670] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:15.202358 13046 tablet_copy_client.cc:538] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:15.213618 13046 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:15.235780 13044 tablet_copy_client.cc:806] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:15.236498 13044 tablet_copy_client.cc:670] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:15.253361 13044 tablet_copy_client.cc:538] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:15.268667 13046 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:15.278458 13046 tablet_bootstrap.cc:492] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:15.279300 13046 ts_tablet_manager.cc:1397] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.066s user 0.015s sys 0.012s
I20250113 08:58:15.281066 12724 raft_consensus.cc:1212] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:15.282460 13046 raft_consensus.cc:357] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.285614 13046 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:15.286288 13046 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.289309 13046 ts_tablet_manager.cc:1428] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20250113 08:58:15.291060 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-705a99b7862446be849d4c843d8084ee received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:15.291556 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-705a99b7862446be849d4c843d8084ee on tablet 705a99b7862446be849d4c843d8084ee with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:15.300541 13044 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:15.337436 13044 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:15.338414 13044 tablet_bootstrap.cc:492] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:15.339138 13044 ts_tablet_manager.cc:1397] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.002s
I20250113 08:58:15.344887 13044 raft_consensus.cc:357] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.345834 13044 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:15.346489 13044 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:15.348765 13044 ts_tablet_manager.cc:1428] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20250113 08:58:15.350805 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-70a6087e1ad14cca8bb3f6d3e0bb1858 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:15.351223 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-70a6087e1ad14cca8bb3f6d3e0bb1858 on tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:15.493571 12724 raft_consensus.cc:1212] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:15.564213 12724 raft_consensus.cc:1212] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:15.713546 12724 raft_consensus.cc:1212] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:15.880738 13061 raft_consensus.cc:1059] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:15.883500 13061 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:15.890077 13059 raft_consensus.cc:1059] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:15.892798 13059 consensus_queue.cc:237] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:15.907509 12724 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:15.912950 11967 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:15.914606 13058 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:15.914891 13060 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:15.915406 12724 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:15.918864 13058 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:15.935793 11890 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:15.937352 12856 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:15.947077 12033 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:15.948937 13061 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:15.952512 11967 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:15.963315 13026 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:15.978514 13022 raft_consensus.cc:1059] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:15.981096 13022 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:15.984175 13056 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:15.991580 12725 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:15.993252 13022 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250113 08:58:15.996352 11884 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:16.000150 13022 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:16.002993 11965 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.004642 11963 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:16.030431 13060 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.035061 11963 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.044453 11787 catalog_manager.cc:5526] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.055752 12724 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.057757 11789 catalog_manager.cc:5526] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250113 08:58:16.071318 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68237.3us
I20250113 08:58:16.093015 12724 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.086095 13022 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20250113 08:58:16.130159 13067 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20250113 08:58:16.130823 12856 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.137049 12041 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.146090 12724 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.211090 13079 raft_consensus.cc:1059] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:16.222368 11965 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:16.222440 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 43798.7us
I20250113 08:58:16.224462 13079 consensus_queue.cc:237] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:16.230748 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 44877.6us
I20250113 08:58:16.242385 11890 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.256705 11893 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.262121 12724 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:16.273586 12041 consensus_queue.cc:237] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:16.279053 13081 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:16.286002 11790 catalog_manager.cc:5526] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.289819 12725 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.292557 11965 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.293849 13085 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:16.295910 12041 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:16.295799 12856 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:16.298997 13060 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:16.300238 11963 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:16.303051 13075 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:16.315943 13056 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.317831 12724 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.332173 11965 raft_consensus.cc:2949] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.349495 13079 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.355597 11965 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.363293 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fcc31099ea8040b7bfbfe7d779148ca5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:16.364773 12034 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.381976 12041 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:16.386668 11788 catalog_manager.cc:5526] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.399549 11963 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.399600 12724 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.404580 13022 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:16.407387 12856 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:16.406978 12725 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.419785 12723 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.427558 11893 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:16.428642 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc31099ea8040b7bfbfe7d779148ca5 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:16.432917 12724 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.416769 13085 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.434557 13075 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:16.440053 11963 raft_consensus.cc:2949] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.447340 12724 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.450102 11790 catalog_manager.cc:5526] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.454247 11788 catalog_manager.cc:5526] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.445760 13079 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.463608 12041 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.482874 13095 tablet_replica.cc:331] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:16.484295 13095 raft_consensus.cc:2238] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:16.485231 13095 raft_consensus.cc:2267] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:16.491024 11790 catalog_manager.cc:5526] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.491986 13095 ts_tablet_manager.cc:1905] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:16.497675 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a99b7862446be849d4c843d8084ee with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:16.506350 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 209260c370164f719208bd1404eef071 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:16.509601 13075 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:16.512573 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 209260c370164f719208bd1404eef071 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:16.514896 13095 ts_tablet_manager.cc:1918] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:16.515579 13095 log.cc:1198] T fcc31099ea8040b7bfbfe7d779148ca5 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/fcc31099ea8040b7bfbfe7d779148ca5
I20250113 08:58:16.518065 13095 tablet_replica.cc:331] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:16.518182 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet fcc31099ea8040b7bfbfe7d779148ca5 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:16.519593 13095 raft_consensus.cc:2238] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:16.520635 13095 raft_consensus.cc:2267] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:16.521351 12041 raft_consensus.cc:2949] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.528287 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc31099ea8040b7bfbfe7d779148ca5 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:16.533047 13095 ts_tablet_manager.cc:1905] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:16.538456 11892 consensus_queue.cc:237] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:16.538952 13096 tablet_replica.cc:331] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:16.541870 13096 raft_consensus.cc:2238] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:16.546283 12724 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.550279 11963 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:16.553313 13075 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:16.555325 13060 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:16.560436 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 705a99b7862446be849d4c843d8084ee with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:16.572219 13081 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.607259 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:16.542693 13096 raft_consensus.cc:2267] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:16.578971 12724 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.579741 11965 raft_consensus.cc:2949] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.622040 13095 ts_tablet_manager.cc:1918] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:16.622731 13095 log.cc:1198] T 705a99b7862446be849d4c843d8084ee P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/705a99b7862446be849d4c843d8084ee
I20250113 08:58:16.650028 13096 ts_tablet_manager.cc:1905] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:16.651031 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 705a99b7862446be849d4c843d8084ee (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:16.670352 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet fcc31099ea8040b7bfbfe7d779148ca5 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:16.648707 11790 catalog_manager.cc:5526] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:16.705375 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:16.717288 13096 ts_tablet_manager.cc:1918] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:16.718020 13096 log.cc:1198] T 209260c370164f719208bd1404eef071 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/209260c370164f719208bd1404eef071
I20250113 08:58:16.721786 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 209260c370164f719208bd1404eef071 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:16.728230 13097 tablet_replica.cc:331] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:16.731791 13097 raft_consensus.cc:2238] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:16.732867 13097 raft_consensus.cc:2267] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:16.736979 13097 ts_tablet_manager.cc:1905] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:16.758774 13097 ts_tablet_manager.cc:1918] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:16.759260 13097 log.cc:1198] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/70a6087e1ad14cca8bb3f6d3e0bb1858
I20250113 08:58:16.761591 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 70a6087e1ad14cca8bb3f6d3e0bb1858 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:17.808559 11890 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:17.817571 11859 consensus_peers.cc:487] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f63a7b00ea546f2865a4b5d730e35de. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:17.819510 11965 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:17.819741 12034 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:17.821285 13075 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:17.822299 13081 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:17.850419 13075 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:17.851776 11963 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:17.863536 12034 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:17.876533 11884 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:17.884951 11787 catalog_manager.cc:5526] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:17.931489 12033 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:17.931983 11859 consensus_peers.cc:487] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a105307f831744ff99c190ff6dd0db1a. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:17.935588 13075 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:17.936420 11963 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:17.938495 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 62184.3us
I20250113 08:58:17.938380 13075 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:17.952200 13075 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:17.957242 12033 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:17.991313 11963 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.005424 11965 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:18.026568 11934 consensus_peers.cc:487] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05d6fc5c56554478ae416a917a542304. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:18.028155 11884 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:18.029788 13086 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:18.029532 11790 catalog_manager.cc:5526] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.031342 12041 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:18.038223 13086 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:18.054489 13108 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.059821 12041 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.074793 11884 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.084091 11788 catalog_manager.cc:5526] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.088546 11890 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.115742 12041 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:18.115911 11859 consensus_peers.cc:487] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2bbae0b61a948339e4c443b9237ad87. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:18.117228 13061 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:18.118054 11965 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:18.120010 13061 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:18.142047 13104 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.143669 12041 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.158983 11965 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.165586 11788 catalog_manager.cc:5526] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:18.328979 13120 ts_tablet_manager.cc:927] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:18.338920 13120 tablet_copy_client.cc:323] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:18.342370 13122 ts_tablet_manager.cc:927] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:18.342535 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet a105307f831744ff99c190ff6dd0db1a from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:18.343116 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet a105307f831744ff99c190ff6dd0db1a from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-a105307f831744ff99c190ff6dd0db1a
I20250113 08:58:18.351552 11903 tablet_copy_source_session.cc:215] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:18.355564 13120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a105307f831744ff99c190ff6dd0db1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:18.360962 13122 tablet_copy_client.cc:323] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:18.363005 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 8f63a7b00ea546f2865a4b5d730e35de from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:18.363579 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 8f63a7b00ea546f2865a4b5d730e35de from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-8f63a7b00ea546f2865a4b5d730e35de
I20250113 08:58:18.374801 11903 tablet_copy_source_session.cc:215] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:18.377880 13122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f63a7b00ea546f2865a4b5d730e35de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:18.384130 13120 tablet_copy_client.cc:806] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:18.384799 13120 tablet_copy_client.cc:670] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:18.388674 13120 tablet_copy_client.cc:538] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:18.399245 13122 tablet_copy_client.cc:806] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:18.399964 13122 tablet_copy_client.cc:670] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:18.406141 13122 tablet_copy_client.cc:538] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:18.411129 13120 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:18.418900 13122 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:18.468797 13122 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:18.469805 13122 tablet_bootstrap.cc:492] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:18.470606 13122 ts_tablet_manager.cc:1397] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.054s user 0.026s sys 0.012s
I20250113 08:58:18.473450 13122 raft_consensus.cc:357] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.475101 13122 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:18.475847 13122 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.483966 13120 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:18.484994 13120 tablet_bootstrap.cc:492] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:18.487650 13120 ts_tablet_manager.cc:1397] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.077s user 0.024s sys 0.014s
I20250113 08:58:18.494935 13120 raft_consensus.cc:357] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.498625 13120 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:18.501571 13120 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.507877 13122 ts_tablet_manager.cc:1428] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.037s user 0.019s sys 0.016s
I20250113 08:58:18.514007 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-8f63a7b00ea546f2865a4b5d730e35de received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:18.514515 13120 ts_tablet_manager.cc:1428] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.026s user 0.010s sys 0.005s
I20250113 08:58:18.514518 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-8f63a7b00ea546f2865a4b5d730e35de on tablet 8f63a7b00ea546f2865a4b5d730e35de with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:18.523270 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a105307f831744ff99c190ff6dd0db1a received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:18.523674 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a105307f831744ff99c190ff6dd0db1a on tablet a105307f831744ff99c190ff6dd0db1a with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:18.533226 13120 ts_tablet_manager.cc:927] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:18.543574 13120 tablet_copy_client.cc:323] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:18.546033 13122 ts_tablet_manager.cc:927] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:18.549580 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet a2bbae0b61a948339e4c443b9237ad87 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:18.550115 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet a2bbae0b61a948339e4c443b9237ad87 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-a2bbae0b61a948339e4c443b9237ad87
I20250113 08:58:18.560120 11903 tablet_copy_source_session.cc:215] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:18.564232 13120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bbae0b61a948339e4c443b9237ad87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:18.567562 13122 tablet_copy_client.cc:323] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:18.571308 11978 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 05d6fc5c56554478ae416a917a542304 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:18.572029 11978 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 05d6fc5c56554478ae416a917a542304 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-05d6fc5c56554478ae416a917a542304
I20250113 08:58:18.586256 11978 tablet_copy_source_session.cc:215] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:18.587764 13120 tablet_copy_client.cc:806] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:18.588596 13120 tablet_copy_client.cc:670] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:18.591578 13122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d6fc5c56554478ae416a917a542304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:18.593230 13120 tablet_copy_client.cc:538] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:18.604871 13120 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:18.623267 13122 tablet_copy_client.cc:806] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:18.624054 13122 tablet_copy_client.cc:670] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:18.634501 13122 tablet_copy_client.cc:538] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:18.674600 13122 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:18.689888 13120 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:18.692234 13120 tablet_bootstrap.cc:492] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:18.693074 13120 ts_tablet_manager.cc:1397] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.088s user 0.039s sys 0.030s
I20250113 08:58:18.701195 13120 raft_consensus.cc:357] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.702171 13120 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:18.702873 13120 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.706598 13120 ts_tablet_manager.cc:1428] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.013s user 0.005s sys 0.003s
I20250113 08:58:18.711498 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a2bbae0b61a948339e4c443b9237ad87 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:18.711988 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a2bbae0b61a948339e4c443b9237ad87 on tablet a2bbae0b61a948339e4c443b9237ad87 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:18.730203 12724 raft_consensus.cc:1212] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:18.731312 13122 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:18.735169 13122 tablet_bootstrap.cc:492] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:18.736086 13122 ts_tablet_manager.cc:1397] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.018s
I20250113 08:58:18.742578 13122 raft_consensus.cc:357] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.743897 13122 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:18.744815 13122 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:18.753814 13122 ts_tablet_manager.cc:1428] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.017s user 0.005s sys 0.005s
I20250113 08:58:18.755824 11978 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-05d6fc5c56554478ae416a917a542304 received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:18.756316 11978 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-05d6fc5c56554478ae416a917a542304 on tablet 05d6fc5c56554478ae416a917a542304 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:18.950994 12725 raft_consensus.cc:1212] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:19.094290 12725 raft_consensus.cc:1212] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:19.156255 12725 raft_consensus.cc:1212] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:19.255304 13075 raft_consensus.cc:1059] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:19.257769 13075 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:19.264401 12725 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.265908 13061 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:19.268352 12041 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.270246 11965 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.272984 13061 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:19.277895 13075 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:19.329051 13061 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:19.332012 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 62475.5us
I20250113 08:58:19.334952 12724 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.344305 11963 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.344640 12035 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.354367 11787 catalog_manager.cc:5526] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.396358 13061 raft_consensus.cc:1059] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:19.400691 13061 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:19.421638 12724 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.423504 11963 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.424010 13075 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:19.426134 12041 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.427676 13104 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:19.429502 13104 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:19.442878 13146 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.453142 12725 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.457770 12035 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.470448 11890 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:19.487497 11963 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:19.489145 13104 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:19.496892 11789 catalog_manager.cc:5526] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.502810 13075 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.500300 11965 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.506237 11967 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.520121 11884 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:19.529810 12725 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:19.541678 13061 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250113 08:58:19.545390 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8f63a7b00ea546f2865a4b5d730e35de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:19.547297 12041 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:19.555589 13075 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:19.569519 13141 raft_consensus.cc:1059] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:19.572192 13141 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:19.572889 13104 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.576542 12034 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:19.594810 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 58470.8us
I20250113 08:58:19.612476 12725 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:19.583571 11788 catalog_manager.cc:5526] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.619388 12725 raft_consensus.cc:2949] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.649349 13150 raft_consensus.cc:1059] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:19.640095 13151 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250113 08:58:19.655381 12724 raft_consensus.cc:2949] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.664239 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f63a7b00ea546f2865a4b5d730e35de with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:19.679406 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a105307f831744ff99c190ff6dd0db1a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:19.703307 13150 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:19.716287 11788 catalog_manager.cc:5526] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:19.720425 12725 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.720849 12041 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.721525 11893 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.743968 13141 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:19.747066 13132 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:19.754220 13153 tablet_replica.cc:331] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:19.756029 13155 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:19.760433 13153 raft_consensus.cc:2238] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:19.777555 13153 raft_consensus.cc:2267] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:19.791319 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet a105307f831744ff99c190ff6dd0db1a with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:19.802876 13153 ts_tablet_manager.cc:1905] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:19.818719 11961 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.821734 13104 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:19.822045 12724 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.824424 13061 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:19.805663 13155 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.814113 12041 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:19.836169 13061 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:19.819049 11884 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.882429 12035 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.869091 13162 tablet_replica.cc:331] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:19.909191 13162 raft_consensus.cc:2238] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:19.913215 13162 raft_consensus.cc:2267] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:19.915351 11790 catalog_manager.cc:5526] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:19.932545 13162 ts_tablet_manager.cc:1905] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250113 08:58:19.927830 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 46153.4us
I20250113 08:58:19.938625 12725 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.941037 13150 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.942652 12723 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.970319 13153 ts_tablet_manager.cc:1918] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:19.970968 13153 log.cc:1198] T 8f63a7b00ea546f2865a4b5d730e35de P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/8f63a7b00ea546f2865a4b5d730e35de
I20250113 08:58:19.979286 12035 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:19.972898 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 8f63a7b00ea546f2865a4b5d730e35de (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:19.996518 13162 ts_tablet_manager.cc:1918] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:19.997232 13162 log.cc:1198] T a105307f831744ff99c190ff6dd0db1a P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/a105307f831744ff99c190ff6dd0db1a
I20250113 08:58:20.005686 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet a105307f831744ff99c190ff6dd0db1a (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:20.012270 11965 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.000187 11788 catalog_manager.cc:5526] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.021165 11967 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:20.027575 12723 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:20.030879 13043 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:20.041364 12041 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:20.044826 13155 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:20.062472 12724 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.056525 13156 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.065544 12041 raft_consensus.cc:2949] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.101128 11884 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:20.116053 12723 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:20.118882 13061 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:20.128041 13075 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.135962 12725 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.154232 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 05d6fc5c56554478ae416a917a542304 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:20.160256 11789 catalog_manager.cc:5526] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.188576 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a2bbae0b61a948339e4c443b9237ad87 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:20.196225 11965 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:20.197453 13146 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:20.202627 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d6fc5c56554478ae416a917a542304 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:20.205615 11790 catalog_manager.cc:5526] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.208132 11961 raft_consensus.cc:2949] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:20.211663 13166 tablet_replica.cc:331] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:20.212682 13166 raft_consensus.cc:2238] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:20.213388 13166 raft_consensus.cc:2267] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:20.220394 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bbae0b61a948339e4c443b9237ad87 with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:20.221578 13153 tablet_replica.cc:331] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:20.221976 13166 ts_tablet_manager.cc:1905] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:20.222975 13153 raft_consensus.cc:2238] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:20.223934 13153 raft_consensus.cc:2267] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:20.229691 13153 ts_tablet_manager.cc:1905] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:20.247886 13166 ts_tablet_manager.cc:1918] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:20.248311 13166 log.cc:1198] T 05d6fc5c56554478ae416a917a542304 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/05d6fc5c56554478ae416a917a542304
I20250113 08:58:20.249650 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 05d6fc5c56554478ae416a917a542304 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:20.257687 13153 ts_tablet_manager.cc:1918] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:20.258217 13153 log.cc:1198] T a2bbae0b61a948339e4c443b9237ad87 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/a2bbae0b61a948339e4c443b9237ad87
I20250113 08:58:20.259754 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet a2bbae0b61a948339e4c443b9237ad87 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:21.258373 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:21.275046 11884 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.281105 12033 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250113 08:58:21.281558 11859 consensus_peers.cc:487] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89d1890dc82d4bd889d0300c9fed050d. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:21.281844 11961 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:21.282826 13167 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.284016 13139 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.304006 13139 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.306658 11965 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.308076 12033 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.326469 11789 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.348104 12034 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.367085 12033 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:21.368009 12033 raft_consensus.cc:604] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:21.370398 12033 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54973520c9884989a12742d4eb45994f"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:21.370826 12033 raft_consensus.cc:604] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:21.383028 11965 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:21.384419 13168 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250113 08:58:21.387501 12008 consensus_peers.cc:487] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b1c553832b94166a0a6089104a74f6c. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:21.402179 11890 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:21.408609 12856 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.405460 13134 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.406507 11961 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.415501 11884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69aa379773c047589ae2362850997a47"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:21.416154 11884 raft_consensus.cc:604] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:21.416410 13168 raft_consensus.cc:988] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:21.416886 13168 raft_consensus.cc:1076] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:21.421511 12034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:21.422737 12034 raft_consensus.cc:604] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:21.423642 11965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:21.424230 11965 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:21.424595 11965 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.426522 12034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "badac2289c8b4a10915cca20182fc423"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:21.427376 12034 raft_consensus.cc:604] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:21.429742 12034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:21.435395 11890 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.438958 12034 raft_consensus.cc:604] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:21.442211 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250113 08:58:21.443982 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:21.443774 11787 catalog_manager.cc:5526] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.444746 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:21.454651 11965 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.455240 11890 consensus_queue.cc:237] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.457315 11965 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:21.457818 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:21.458446 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:21.458545 12033 raft_consensus.cc:3049] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:21.459007 11884 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.459182 12033 raft_consensus.cc:738] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:21.459947 12033 consensus_queue.cc:260] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.460970 12033 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:58:21.470178 11859 consensus_peers.cc:487] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49d70ffc378340e785c4f8dde909f9ba. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:21.470818 12041 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:21.474164 13167 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.475347 11884 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:21.476807 11965 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:21.478158 13167 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.479241 11932 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:21.482098 13172 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:21.482537 13172 raft_consensus.cc:695] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:21.484055 13172 consensus_queue.cc:237] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.486001 12033 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:21.499284 13139 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.503886 11961 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.507737 11789 catalog_manager.cc:5526] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:21.511039 12034 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.533370 11789 catalog_manager.cc:5526] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.540128 11961 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.552644 12034 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:21.552750 11934 consensus_peers.cc:487] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a457bc5206c46d7b198dbff2b1c137c. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:21.554172 13172 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.554505 11890 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:21.555706 13156 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.563835 13156 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.564989 11890 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.570103 12034 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.590734 11789 catalog_manager.cc:5526] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:21.630021 13182 raft_consensus.cc:988] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:21.630440 13134 raft_consensus.cc:1076] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:21.631871 12723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "badac2289c8b4a10915cca20182fc423"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38520
I20250113 08:58:21.632431 12723 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:21.632791 12723 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.639763 12723 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.644238 12723 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:21.647697 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:21.648309 12041 raft_consensus.cc:3049] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:21.644575 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:21.648612 12041 raft_consensus.cc:738] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:21.648810 11965 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.649350 12041 consensus_queue.cc:260] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.650337 12041 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.655771 11965 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:21.656337 12041 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:21.656754 12686 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:21.658564 13193 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:21.659898 13193 raft_consensus.cc:695] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:21.660809 13193 consensus_queue.cc:237] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.673576 11790 catalog_manager.cc:5526] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:21.698886 13196 ts_tablet_manager.cc:927] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:21.700846 13196 tablet_copy_client.cc:323] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:21.702239 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 89d1890dc82d4bd889d0300c9fed050d from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:21.702668 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 89d1890dc82d4bd889d0300c9fed050d from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-89d1890dc82d4bd889d0300c9fed050d
I20250113 08:58:21.712524 11903 tablet_copy_source_session.cc:215] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:21.715317 13196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89d1890dc82d4bd889d0300c9fed050d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:21.728754 13196 tablet_copy_client.cc:806] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:21.729310 13196 tablet_copy_client.cc:670] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:21.732869 13196 tablet_copy_client.cc:538] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:21.743733 13196 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:21.757683 13134 raft_consensus.cc:988] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:21.758216 13134 raft_consensus.cc:1076] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:21.759712 11965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:21.760329 11965 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:21.760727 11965 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.762779 13196 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:21.763697 13196 tablet_bootstrap.cc:492] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:21.764338 13196 ts_tablet_manager.cc:1397] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.002s
I20250113 08:58:21.767232 13196 raft_consensus.cc:357] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.768182 13196 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:21.768895 13196 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.771551 13196 ts_tablet_manager.cc:1428] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:58:21.773188 13199 ts_tablet_manager.cc:927] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:21.777544 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-89d1890dc82d4bd889d0300c9fed050d received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:21.778124 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-89d1890dc82d4bd889d0300c9fed050d on tablet 89d1890dc82d4bd889d0300c9fed050d with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:21.782910 13199 tablet_copy_client.cc:323] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:21.784504 13146 raft_consensus.cc:988] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:21.785002 13075 raft_consensus.cc:1076] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:21.786449 12724 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69aa379773c047589ae2362850997a47"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38504
I20250113 08:58:21.786844 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 8b1c553832b94166a0a6089104a74f6c from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:21.786957 12724 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:21.787047 11965 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.787492 12724 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.787446 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 8b1c553832b94166a0a6089104a74f6c from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-8b1c553832b94166a0a6089104a74f6c
I20250113 08:58:21.789561 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:21.790092 11890 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.794971 12724 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.797492 12724 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:21.798854 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:21.799348 11884 raft_consensus.cc:3049] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:21.799665 11884 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:21.800233 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:21.800755 12034 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.800962 11884 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.802868 12052 tablet_copy_source_session.cc:215] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:21.806365 11884 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.808444 12034 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:21.811668 13199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1c553832b94166a0a6089104a74f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:21.816817 12684 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:21.818069 13193 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:21.818554 13193 raft_consensus.cc:695] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:21.819160 13193 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.814000 11890 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:21.827967 11965 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:21.815017 11884 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:21.828866 11932 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:21.829788 13141 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:21.830251 13141 raft_consensus.cc:695] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:21.831122 13141 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.836616 11790 catalog_manager.cc:5526] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:21.837798 13199 tablet_copy_client.cc:806] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:21.838622 13199 tablet_copy_client.cc:670] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:21.828862 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:21.842532 12034 raft_consensus.cc:3049] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:21.842818 12034 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:21.843436 12034 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.844254 12034 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.845818 13199 tablet_copy_client.cc:538] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:21.854101 12034 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:21.856139 13199 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:21.868005 13134 raft_consensus.cc:988] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:21.868490 13134 raft_consensus.cc:1076] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:21.870318 12724 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38520
I20250113 08:58:21.870934 12724 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:21.871352 12724 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.891536 12724 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.894425 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:21.895020 11965 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.898577 11790 catalog_manager.cc:5526] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:21.902172 11965 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:21.923944 12686 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:21.925202 13193 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:21.925755 13193 raft_consensus.cc:695] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:21.925813 12724 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:21.923933 13168 raft_consensus.cc:988] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:21.926834 13168 raft_consensus.cc:1076] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:21.926626 13193 consensus_queue.cc:237] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.926237 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:21.929617 11965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54973520c9884989a12742d4eb45994f"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:21.930150 11965 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:21.930476 11965 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.925177 13199 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:21.935461 13199 tablet_bootstrap.cc:492] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:21.936470 11965 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.928129 12034 raft_consensus.cc:3049] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:21.937899 13199 ts_tablet_manager.cc:1397] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.082s user 0.029s sys 0.011s
I20250113 08:58:21.937862 12034 raft_consensus.cc:738] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:21.938824 12034 consensus_queue.cc:260] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:21.940150 12034 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.941143 11890 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:21.942744 13199 raft_consensus.cc:357] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.943547 13199 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:21.944298 13141 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:21.946070 12034 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:21.952394 11965 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:21.955606 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:21.956210 12041 raft_consensus.cc:3049] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:21.956496 12041 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:21.944145 13199 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:21.950784 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:21.957158 12041 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.957842 11884 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.958648 12041 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:21.959429 13199 ts_tablet_manager.cc:1428] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250113 08:58:21.958592 11790 catalog_manager.cc:5526] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:21.961009 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-8b1c553832b94166a0a6089104a74f6c received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:21.961421 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-8b1c553832b94166a0a6089104a74f6c on tablet 8b1c553832b94166a0a6089104a74f6c with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:21.966089 12041 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:21.967419 11931 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:21.968175 13141 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:21.968631 13141 raft_consensus.cc:695] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:21.969501 13141 consensus_queue.cc:237] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:21.971319 11884 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:21.995802 12034 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:21.997112 13192 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:21.997753 11787 catalog_manager.cc:5526] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:22.027513 13199 ts_tablet_manager.cc:927] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:22.043267 13199 tablet_copy_client.cc:323] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:22.044759 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 49d70ffc378340e785c4f8dde909f9ba from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:22.045382 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 49d70ffc378340e785c4f8dde909f9ba from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-49d70ffc378340e785c4f8dde909f9ba
I20250113 08:58:22.053639 11903 tablet_copy_source_session.cc:215] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:22.056595 13199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d70ffc378340e785c4f8dde909f9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:22.070044 13199 tablet_copy_client.cc:806] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:22.070569 13199 tablet_copy_client.cc:670] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:22.075232 13199 tablet_copy_client.cc:538] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:22.080041 13196 ts_tablet_manager.cc:927] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:22.083428 13196 tablet_copy_client.cc:323] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:22.084879 11978 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 9a457bc5206c46d7b198dbff2b1c137c from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:22.085425 11978 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 9a457bc5206c46d7b198dbff2b1c137c from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-9a457bc5206c46d7b198dbff2b1c137c
I20250113 08:58:22.088305 13199 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:22.091189 11978 tablet_copy_source_session.cc:215] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:22.094020 13196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a457bc5206c46d7b198dbff2b1c137c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:22.110038 13199 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:22.110349 13196 tablet_copy_client.cc:806] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:22.111002 13196 tablet_copy_client.cc:670] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:22.111179 13199 tablet_bootstrap.cc:492] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:22.113454 13199 ts_tablet_manager.cc:1397] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.010s
I20250113 08:58:22.114400 12724 raft_consensus.cc:1212] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250113 08:58:22.116346 13199 raft_consensus.cc:357] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:22.117439 13199 raft_consensus.cc:738] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:22.118050 13199 consensus_queue.cc:260] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:22.122519 13199 ts_tablet_manager.cc:1428] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20250113 08:58:22.123963 13196 tablet_copy_client.cc:538] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:22.124894 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-49d70ffc378340e785c4f8dde909f9ba received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:22.125406 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-49d70ffc378340e785c4f8dde909f9ba on tablet 49d70ffc378340e785c4f8dde909f9ba with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:22.125840 11965 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:22.129520 13193 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:22.133989 13196 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:22.140784 12041 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:22.146200 13193 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:22.213577 12724 raft_consensus.cc:1212] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:22.242789 12034 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:22.245393 13156 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:22.246837 11890 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:22.249516 13193 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:22.272925 13196 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:22.277949 11890 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:22.284698 13196 tablet_bootstrap.cc:492] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:22.285691 13196 ts_tablet_manager.cc:1397] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.152s user 0.063s sys 0.043s
I20250113 08:58:22.288338 13156 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:22.296021 12041 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:22.298225 13218 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:22.317270 13196 raft_consensus.cc:357] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:22.321861 13196 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:22.322851 13196 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:22.328522 13196 ts_tablet_manager.cc:1428] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.042s user 0.006s sys 0.001s
I20250113 08:58:22.333832 11978 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-9a457bc5206c46d7b198dbff2b1c137c received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:22.334314 11978 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-9a457bc5206c46d7b198dbff2b1c137c on tablet 9a457bc5206c46d7b198dbff2b1c137c with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:22.395965 12033 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:22.397559 13194 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:22.411592 11967 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:22.413884 13194 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:22.496119 11884 raft_consensus.cc:1270] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:22.497403 13192 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:22.518915 12041 raft_consensus.cc:1270] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:22.520678 13156 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:22.549669 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:22.553371 12724 raft_consensus.cc:1212] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:22.584865 12724 raft_consensus.cc:1212] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:22.608767 13168 raft_consensus.cc:1059] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:22.611436 13168 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:22.625175 13146 raft_consensus.cc:1059] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:22.629637 13146 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:22.674834 12724 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:22.686112 13182 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:22.736905 12041 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:22.739775 13075 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:22.788941 11892 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:22.820031 13233 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:22.885118 13233 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:22.895376 11890 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:22.919073 11963 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:22.925120 11963 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:22.947811 12724 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:22.952768 13139 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:22.958582 12723 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:22.982417 13075 raft_consensus.cc:1059] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:22.984874 13075 consensus_queue.cc:237] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.015506 13172 raft_consensus.cc:1059] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:23.020578 13172 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:23.066174 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 51566.3us
I20250113 08:58:23.070719 12035 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:23.071480 12034 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:23.087544 11963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05d6fc5c56554478ae416a917a542304"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:23.089391 11963 raft_consensus.cc:604] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:23.083133 12723 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:23.101796 13227 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.106240 11884 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:23.113343 13238 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.115723 13239 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.117290 11788 catalog_manager.cc:5526] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:23.121363 13192 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.126920 11967 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:23.143508 12723 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:23.151731 13245 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.175632 11893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:23.176294 11893 raft_consensus.cc:604] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:23.178000 13245 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.179468 13139 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:23.195194 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 94364.7us
I20250113 08:58:23.198051 11967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:23.205334 13168 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.204020 12725 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.199121 11967 raft_consensus.cc:604] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:23.235450 13146 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.238426 11890 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:23.276489 11890 raft_consensus.cc:604] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:23.286500 11962 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:23.287326 11962 raft_consensus.cc:604] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:23.294648 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250113 08:58:23.301968 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:23.315697 12034 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.387616 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:23.398270 13237 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.417207 11890 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.420369 13075 raft_consensus.cc:988] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:23.420878 13075 raft_consensus.cc:1076] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:23.439851 12725 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.441320 11964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:23.442225 11964 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:23.443126 11964 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.456831 11960 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.477955 11964 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.481021 11964 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:23.481398 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:23.483772 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:23.484404 12037 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.488718 13192 raft_consensus.cc:988] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:23.489368 13172 raft_consensus.cc:1076] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:23.491112 12723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05d6fc5c56554478ae416a917a542304"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38492
I20250113 08:58:23.491695 12723 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:23.492086 12723 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.492439 12037 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:23.482239 11891 raft_consensus.cc:3049] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:23.503389 11891 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:23.504712 11891 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.510010 11891 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.499923 13146 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.516270 12041 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.519016 13242 raft_consensus.cc:988] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:23.519532 13242 raft_consensus.cc:1076] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:23.524322 11960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:23.524971 11960 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:23.525429 11960 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:23.525110 12725 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.533604 11967 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.550323 11931 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:23.559866 13156 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:23.560424 13156 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:23.561527 13156 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.613502 11960 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.515995 12723 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.626528 11965 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:23.628434 13192 raft_consensus.cc:988] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:23.628789 11787 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.632097 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:23.635491 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:23.636174 11960 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:23.640600 11890 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:23.640681 12725 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:23.642129 11890 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:23.640249 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:23.641086 11891 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:23.643512 12037 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:23.643288 13239 raft_consensus.cc:1076] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:23.647094 11787 catalog_manager.cc:5526] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.653512 12041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33878
W20250113 08:58:23.653983 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 92095.9us
I20250113 08:58:23.654098 12041 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:23.654475 13239 raft_consensus.cc:988] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:23.654950 13239 raft_consensus.cc:1076] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:23.654505 12041 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.657392 12034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33878
I20250113 08:58:23.657936 12034 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:23.658248 12034 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.649978 11787 catalog_manager.cc:5526] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.661671 11890 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.664307 13261 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:23.666280 12725 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:23.666923 12033 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.670961 11890 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:23.674067 12037 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:23.675160 12041 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.679044 11931 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:23.679955 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:23.680964 11960 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:23.681363 11960 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:23.679982 13192 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:23.683352 13192 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:23.683507 12725 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:23.683218 11960 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.684331 11960 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.676307 12034 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.685106 12041 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:23.687717 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:23.688170 11892 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:23.688359 11893 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.689273 13134 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250113 08:58:23.693245 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:23.693992 11965 raft_consensus.cc:3049] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:23.692833 12042 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.693833 13134 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:23.686311 13192 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.706420 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:23.707062 12033 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.699128 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:23.700661 12723 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:23.699154 11790 catalog_manager.cc:5526] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:23.701128 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:23.710101 11967 raft_consensus.cc:3049] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:23.710402 11967 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:23.711941 12034 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:23.708556 13246 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.711010 11967 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.714218 11967 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.716226 11891 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.709601 11965 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:23.702929 11890 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:23.716363 12725 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.721642 11965 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.722319 11967 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:23.722975 11965 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:23.723662 12686 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:23.724731 13193 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:23.727917 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8b1c553832b94166a0a6089104a74f6c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:23.725268 13193 raft_consensus.cc:695] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:23.732146 13193 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.733091 11960 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:23.748584 12007 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:23.755963 13134 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:23.756436 13134 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:23.757231 13134 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.760865 11890 consensus_queue.cc:237] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.749524 11789 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 2, term changed from 2 to 3, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from VOTER to NON_VOTER, now has a pending config: VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129), VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130), VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131), VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.768015 12726 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:23.769495 13261 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:23.772593 11962 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:23.773345 11890 raft_consensus.cc:2949] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.748469 11893 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:23.780675 11790 catalog_manager.cc:5526] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250113 08:58:23.783429 11989 debug-util.cc:398] Leaking SignalData structure 0x7b0800148d40 after lost signal to thread 10705
I20250113 08:58:23.783311 13262 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:23.784885 11989 debug-util.cc:398] Leaking SignalData structure 0x7b080016b740 after lost signal to thread 11825
W20250113 08:58:23.786382 11989 debug-util.cc:398] Leaking SignalData structure 0x7b0800316400 after lost signal to thread 11917
W20250113 08:58:23.793651 11989 debug-util.cc:398] Leaking SignalData structure 0x7b08002ccbc0 after lost signal to thread 11992
W20250113 08:58:23.794996 11989 debug-util.cc:398] Leaking SignalData structure 0x7b080016d1c0 after lost signal to thread 12066
W20250113 08:58:23.796326 11989 debug-util.cc:398] Leaking SignalData structure 0x7b0800312f20 after lost signal to thread 12750
I20250113 08:58:23.796754 12726 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.754326 11967 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:23.806295 13261 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:23.809111 11891 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:23.811570 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49d70ffc378340e785c4f8dde909f9ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:23.830974 11965 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:23.833065 12006 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:23.835220 12033 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:23.782819 11787 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 2 to 3, term changed from 3 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129), VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130), VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131), VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.838481 11960 raft_consensus.cc:2949] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.841773 13258 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:23.842320 13258 raft_consensus.cc:695] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:23.845432 11962 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:23.846068 11962 raft_consensus.cc:604] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Received request to transfer leadership
I20250113 08:58:23.847985 13258 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:23.851749 11790 catalog_manager.cc:5526] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.854245 11789 catalog_manager.cc:5526] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:23.862056 12726 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:23.863844 13192 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:23.866214 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1c553832b94166a0a6089104a74f6c with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:23.871551 12034 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:23.873222 13156 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:23.888885 13239 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.897240 12034 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.930852 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9a457bc5206c46d7b198dbff2b1c137c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:23.937562 12725 raft_consensus.cc:2949] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:23.946589 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d70ffc378340e785c4f8dde909f9ba with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:23.955462 11789 catalog_manager.cc:5526] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:23.962092 11790 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:23.963832 13267 tablet_replica.cc:331] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:23.965198 13267 raft_consensus.cc:2238] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:23.966064 13267 raft_consensus.cc:2267] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:23.976223 11790 catalog_manager.cc:5526] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:23.981948 13267 ts_tablet_manager.cc:1905] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:23.989419 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a457bc5206c46d7b198dbff2b1c137c with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:23.992173 13268 tablet_replica.cc:331] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:23.993680 13268 raft_consensus.cc:2238] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:23.996047 13268 raft_consensus.cc:2267] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:24.007182 13268 ts_tablet_manager.cc:1905] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:24.013098 13269 tablet_replica.cc:331] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:24.014839 13269 raft_consensus.cc:2238] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:24.015738 13269 raft_consensus.cc:2267] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:24.028335 13269 ts_tablet_manager.cc:1905] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:24.048705 13267 ts_tablet_manager.cc:1918] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:24.049304 13267 log.cc:1198] T 8b1c553832b94166a0a6089104a74f6c P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/8b1c553832b94166a0a6089104a74f6c
I20250113 08:58:24.052464 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 8b1c553832b94166a0a6089104a74f6c (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:24.052958 13268 ts_tablet_manager.cc:1918] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:24.053429 13268 log.cc:1198] T 49d70ffc378340e785c4f8dde909f9ba P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/49d70ffc378340e785c4f8dde909f9ba
I20250113 08:58:24.066085 13269 ts_tablet_manager.cc:1918] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:24.067052 13269 log.cc:1198] T 9a457bc5206c46d7b198dbff2b1c137c P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/9a457bc5206c46d7b198dbff2b1c137c
I20250113 08:58:24.071858 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 9a457bc5206c46d7b198dbff2b1c137c (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:24.073493 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 49d70ffc378340e785c4f8dde909f9ba (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:24.121801 11891 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250113 08:58:24.122908 13192 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:24.136065 11790 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 2 to 3, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:24.137831 13192 raft_consensus.cc:988] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:24.138316 13239 raft_consensus.cc:1076] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:24.139683 11891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:24.140111 11891 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:24.140420 11891 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:24.147902 11891 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:24.148589 12725 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250113 08:58:24.149982 13239 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:24.151298 11891 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:24.154331 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:24.154875 12722 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:24.158313 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:24.158867 12034 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:24.164237 11962 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:24.165729 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:24.161324 13239 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:24.174253 12722 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:24.174330 12033 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:24.177284 13141 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:24.185920 12034 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:24.187173 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:24.188162 13227 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:24.188534 13227 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:24.189340 13227 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:24.195957 12725 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:24.205312 11891 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:24.206583 13192 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:24.209949 11893 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:24.209785 13168 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:24.205152 11790 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 3 to 4, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 4 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:24.211907 13141 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:24.256484 12033 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:24.261871 13141 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:24.263767 11967 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:24.265959 13168 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:24.273073 12033 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:24.276665 11964 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:24.277731 13194 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:24.280848 13168 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250113 08:58:24.289620 11964 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:24.296826 13194 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:24.322618 11891 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:24.332813 13168 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:24.333294 13239 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:24.333837 13239 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:24.334275 11891 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:24.337975 13238 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:24.338665 13238 consensus_queue.cc:1225] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20250113 08:58:24.358129 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 56258.3us
I20250113 08:58:24.378057 12033 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20250113 08:58:24.383546 13239 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:24.427899 13192 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250113 08:58:24.500030 13239 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:24.485356 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 103079us
I20250113 08:58:24.590992 12725 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250113 08:58:24.603523 13227 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.009s
I20250113 08:58:24.659230 12034 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250113 08:58:24.661499 13227 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:24.669322 11962 raft_consensus.cc:604] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Received request to transfer leadership
I20250113 08:58:24.669762 11962 raft_consensus.cc:612] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250113 08:58:24.680194 11839 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
W20250113 08:58:24.680725 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20250113 08:58:24.712466 11960 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:58:24.713513 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 241565us
I20250113 08:58:24.715493 11962 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250113 08:58:24.718581 13146 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:24.823060 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:24.841878 11891 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:24.849087 12034 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:24.850754 13261 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:24.849138 12722 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:24.854699 13261 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:24.858992 13245 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:24.869000 12034 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:24.868819 12722 raft_consensus.cc:2949] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:24.883198 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 89d1890dc82d4bd889d0300c9fed050d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:24.893937 11787 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 4 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:24.895859 11960 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:24.897707 11960 raft_consensus.cc:604] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:24.900491 11892 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:24.901036 11892 raft_consensus.cc:604] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:24.903764 12042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:24.904313 12042 raft_consensus.cc:604] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:24.907881 12042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:24.908521 12042 raft_consensus.cc:604] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:24.911294 11892 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:24.911904 11892 raft_consensus.cc:604] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:24.913749 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250113 08:58:24.916795 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:24.917500 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:24.978250 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 89d1890dc82d4bd889d0300c9fed050d with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:24.982892 13245 raft_consensus.cc:988] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:24.983402 13245 raft_consensus.cc:1076] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:24.984876 12034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33876
I20250113 08:58:24.989147 12034 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:24.989542 12034 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:24.994903 13302 tablet_replica.cc:331] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:24.996019 13302 raft_consensus.cc:2238] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:24.996767 13302 raft_consensus.cc:2267] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:24.999074 12034 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:25.002527 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:25.002661 12034 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:25.004308 11892 raft_consensus.cc:3049] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:25.004643 11892 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:25.005896 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:25.006510 11964 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.005401 11892 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:25.011683 11892 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.014282 11964 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:25.015543 12007 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:25.016515 13258 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:25.017005 13258 raft_consensus.cc:695] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:25.017875 13258 consensus_queue.cc:237] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:25.019141 13302 ts_tablet_manager.cc:1905] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:25.021128 11892 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:25.030025 11789 catalog_manager.cc:5526] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:25.036867 13302 ts_tablet_manager.cc:1918] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20250113 08:58:25.037473 13302 log.cc:1198] T 89d1890dc82d4bd889d0300c9fed050d P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/89d1890dc82d4bd889d0300c9fed050d
I20250113 08:58:25.039129 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 89d1890dc82d4bd889d0300c9fed050d (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:25.174508 13279 raft_consensus.cc:988] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:25.175140 13279 raft_consensus.cc:1076] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:25.177834 12726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38520
I20250113 08:58:25.178436 12726 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:25.178776 12726 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.190384 12726 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.193291 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:25.193970 11891 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.194243 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:25.194908 12034 raft_consensus.cc:3049] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:25.195194 12034 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:25.193017 12726 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:25.195989 12034 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.200852 12034 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.201687 11891 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:25.205933 12685 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:25.206866 13297 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:25.207324 13297 raft_consensus.cc:695] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:25.208171 13297 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.213721 12034 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:25.215771 13146 raft_consensus.cc:988] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:25.216238 13227 raft_consensus.cc:1076] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:25.217710 12726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38504
I20250113 08:58:25.218254 12726 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:25.218619 12726 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.220829 11789 catalog_manager.cc:5526] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:25.226648 12726 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.229395 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:25.229785 12726 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:25.229979 11891 raft_consensus.cc:3049] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:25.230351 11891 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:25.231086 11891 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.232406 11891 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.232690 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:25.236882 12034 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.241539 11891 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:25.242506 12685 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:25.243281 13297 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:25.243696 13297 raft_consensus.cc:695] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:25.244532 13297 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.245119 12034 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:25.262856 11789 catalog_manager.cc:5526] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:25.311307 13292 raft_consensus.cc:988] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:25.312171 13239 raft_consensus.cc:1076] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:25.314314 12726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38492
I20250113 08:58:25.314836 12726 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:25.315099 12726 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.321703 12726 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.323215 12726 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:25.324334 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:25.325182 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:25.326170 11964 raft_consensus.cc:3049] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:25.326532 11964 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:25.327407 11964 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.325874 11891 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.329077 11964 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:25.337340 11964 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:25.338577 12686 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:25.351857 13286 raft_consensus.cc:988] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:25.352370 13279 raft_consensus.cc:1076] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:25.355149 11964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:25.355819 11964 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:25.356194 11964 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:25.357182 13297 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:25.357744 13297 raft_consensus.cc:695] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:25.358580 13297 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:25.371083 11891 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:25.377063 11964 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:25.378355 11789 catalog_manager.cc:5526] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:25.380467 11964 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:25.381384 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:25.382019 12042 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:25.381992 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:25.382529 12042 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:25.382740 11891 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:25.383337 12042 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:25.384665 12042 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:25.390190 11891 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:25.391417 11932 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:25.392185 13239 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:25.392769 13239 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:25.393396 12042 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:25.393839 13239 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:25.404392 11789 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 2 to 3, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:25.422475 11891 raft_consensus.cc:1270] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:25.423686 13258 consensus_queue.cc:1035] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:25.434414 11964 raft_consensus.cc:1270] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:25.436102 13258 consensus_queue.cc:1035] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:25.647118 12034 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:25.648342 13297 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:25.663673 11884 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:25.669847 13297 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:25.695282 12042 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:25.696920 13297 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:25.720993 11891 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:25.724179 13314 consensus_queue.cc:1035] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:25.803217 11964 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:25.804428 13314 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:25.807403 12034 consensus_queue.cc:237] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:25.814746 11884 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250113 08:58:25.816007 12008 consensus_peers.cc:487] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80cc6c97986444348b1837c27c41338a. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:25.816001 13292 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:25.818711 11960 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:25.819904 13279 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:25.826148 11891 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:25.828284 13289 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:25.833071 11892 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:25.834776 13279 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:25.921782 13258 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:25.927863 12034 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:25.929175 13294 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:25.934943 11892 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:25.978843 11892 consensus_queue.cc:237] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:25.980801 11790 catalog_manager.cc:5526] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:25.983383 11964 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.000641 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:26.026093 12034 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:26.027756 13261 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250113 08:58:26.030786 11859 consensus_peers.cc:487] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c3345ca9b1640a4b4e6d2efd2ee061d. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:26.060489 11964 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:26.063060 13261 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:26.071736 12034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:26.072391 12034 raft_consensus.cc:604] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:26.073045 13227 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.081446 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:26.082094 11891 raft_consensus.cc:604] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:26.090651 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "992519c5663649019a1ff9753075ddc7"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:26.091297 11891 raft_consensus.cc:604] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:26.091246 11960 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.080749 12042 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.098450 11964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:26.099016 11964 raft_consensus.cc:604] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:26.102049 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:26.102663 11891 raft_consensus.cc:604] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:26.119307 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250113 08:58:26.120996 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:26.121601 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:26.127252 11891 consensus_queue.cc:237] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:26.123557 11789 catalog_manager.cc:5526] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.135329 13146 raft_consensus.cc:988] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:26.142418 13146 raft_consensus.cc:1076] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:26.157280 11960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:26.157986 11960 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:26.158390 11960 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.164948 11960 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.166958 13245 raft_consensus.cc:988] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:26.167488 13335 raft_consensus.cc:1076] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:26.169589 11964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:26.169812 11960 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:26.170099 11964 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:26.170469 11964 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.170773 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:26.171311 12037 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.173915 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:26.174515 11884 raft_consensus.cc:3049] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:26.174784 11884 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:26.175468 11884 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.176515 11884 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.177907 12037 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:26.179235 11931 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:26.180300 13239 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:26.180799 13239 raft_consensus.cc:695] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:26.181716 13239 consensus_queue.cc:237] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.184545 11884 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:26.217752 11790 catalog_manager.cc:5526] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:26.251603 11964 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:26.260800 12037 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:26.263396 13227 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:26.264452 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:26.264917 11859 consensus_peers.cc:487] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a6bba22615448a8acbbb898fb381116. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:26.265107 11888 raft_consensus.cc:3049] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:26.265518 11888 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:26.266337 11888 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:26.267596 11888 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.270186 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:26.270776 12726 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.273532 13245 raft_consensus.cc:988] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:26.274021 13146 raft_consensus.cc:1076] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:26.275673 12042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "992519c5663649019a1ff9753075ddc7"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33876
I20250113 08:58:26.276293 12042 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:26.276697 12042 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.280417 11888 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:26.282305 11964 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:26.281240 12726 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:26.283367 11963 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:26.283811 11934 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:26.284862 13227 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:26.287838 13325 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:26.288336 13325 raft_consensus.cc:695] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:26.300907 13325 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:26.310488 13326 raft_consensus.cc:988] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:26.310981 13326 raft_consensus.cc:1076] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:26.312736 12726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38520
I20250113 08:58:26.313241 12726 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:26.313597 12726 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.316093 12042 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.318766 13245 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.322108 12726 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:26.324527 12726 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:26.325203 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:26.325920 12037 raft_consensus.cc:3049] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:26.326222 12037 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:26.327013 12037 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:26.328346 12037 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.330539 12033 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.331390 11963 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.332541 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:26.333149 11964 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.340114 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:26.340768 11891 raft_consensus.cc:3049] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:26.341061 11891 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:26.343772 12042 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:26.345983 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:26.346601 11967 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.341758 11891 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.354375 11891 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:26.356308 11789 catalog_manager.cc:5526] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:26.361440 13292 raft_consensus.cc:988] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:26.362210 11888 consensus_queue.cc:237] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:26.368074 13292 raft_consensus.cc:1076] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:26.364022 11790 catalog_manager.cc:5526] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:26.369606 12033 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33878
I20250113 08:58:26.370146 12033 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:26.370487 12033 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:26.378392 12033 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.379339 12037 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:26.381476 12033 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:26.383059 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:26.383744 11960 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:26.384052 11960 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:26.384187 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:26.384840 11884 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:26.386960 11964 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:26.390870 12684 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:26.391754 13289 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:26.392210 13289 raft_consensus.cc:695] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:26.393051 13289 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:26.397897 13341 ts_tablet_manager.cc:927] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:26.384855 11960 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.404978 11960 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:26.402773 11967 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:26.407191 12007 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:26.409229 13326 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:26.409910 13326 raft_consensus.cc:695] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:26.413763 11884 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:26.418040 13326 consensus_queue.cc:237] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.417382 11960 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:26.421064 12007 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:26.424170 11891 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:26.424257 13258 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:26.431545 13258 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:26.433988 13341 tablet_copy_client.cc:323] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:26.432485 13258 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:26.444904 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 80cc6c97986444348b1837c27c41338a from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:26.445587 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 80cc6c97986444348b1837c27c41338a from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-80cc6c97986444348b1837c27c41338a
W20250113 08:58:26.465967 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 55769.1us
I20250113 08:58:26.480837 12052 tablet_copy_source_session.cc:215] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:26.498157 11789 catalog_manager.cc:5526] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:26.513950 13341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc6c97986444348b1837c27c41338a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250113 08:58:26.514232 11859 consensus_peers.cc:487] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a9d15ff319f43f0a1dd5ae717393675. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:26.516260 11960 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:26.518003 13227 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:26.511983 11787 catalog_manager.cc:5526] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:26.520444 12037 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:26.522279 13227 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:26.544220 13341 tablet_copy_client.cc:806] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:26.544865 13341 tablet_copy_client.cc:670] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:26.551044 13341 tablet_copy_client.cc:538] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:26.550360 13245 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.558830 11960 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.571018 13351 ts_tablet_manager.cc:927] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:26.571615 11891 raft_consensus.cc:1270] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:26.570452 13341 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:26.581534 13345 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:26.594339 12034 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.635020 13351 tablet_copy_client.cc:323] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:26.636955 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:26.637569 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 2c3345ca9b1640a4b4e6d2efd2ee061d from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-2c3345ca9b1640a4b4e6d2efd2ee061d
I20250113 08:58:26.647207 11903 tablet_copy_source_session.cc:215] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:26.661408 13351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:26.665967 11787 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 4, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:26.710078 13351 tablet_copy_client.cc:806] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:26.708395 11787 catalog_manager.cc:5526] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:26.710815 13351 tablet_copy_client.cc:670] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:26.721491 13351 tablet_copy_client.cc:538] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:26.748525 13351 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:26.797030 13358 ts_tablet_manager.cc:927] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:26.803041 13358 tablet_copy_client.cc:323] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:26.804597 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 9a6bba22615448a8acbbb898fb381116 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:26.805122 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 9a6bba22615448a8acbbb898fb381116 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-9a6bba22615448a8acbbb898fb381116
I20250113 08:58:26.813560 12034 raft_consensus.cc:1270] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:26.815567 13325 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:26.824678 11903 tablet_copy_source_session.cc:215] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:26.834748 13358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6bba22615448a8acbbb898fb381116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:26.869666 13341 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:26.870951 13341 tablet_bootstrap.cc:492] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:26.871791 13341 ts_tablet_manager.cc:1397] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.302s user 0.079s sys 0.120s
I20250113 08:58:26.873404 11884 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:26.874994 13358 tablet_copy_client.cc:806] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:26.874868 13325 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:26.875767 13358 tablet_copy_client.cc:670] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:26.879467 13351 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:26.880501 13351 tablet_bootstrap.cc:492] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:26.880832 13358 tablet_copy_client.cc:538] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:26.881183 13351 ts_tablet_manager.cc:1397] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.133s user 0.018s sys 0.013s
I20250113 08:58:26.875128 13341 raft_consensus.cc:357] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:26.892642 13341 raft_consensus.cc:738] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:26.893827 13341 consensus_queue.cc:260] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:26.897076 13341 ts_tablet_manager.cc:1428] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250113 08:58:26.886498 13351 raft_consensus.cc:357] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:26.898276 13351 raft_consensus.cc:738] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:26.899166 13351 consensus_queue.cc:260] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:26.902102 13351 ts_tablet_manager.cc:1428] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250113 08:58:26.909813 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-2c3345ca9b1640a4b4e6d2efd2ee061d received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:26.910307 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-2c3345ca9b1640a4b4e6d2efd2ee061d on tablet 2c3345ca9b1640a4b4e6d2efd2ee061d with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:26.917903 12722 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:26.919661 13358 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:26.919828 13294 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:26.924059 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-80cc6c97986444348b1837c27c41338a received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:26.924484 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-80cc6c97986444348b1837c27c41338a on tablet 80cc6c97986444348b1837c27c41338a with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:26.937533 11960 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:26.939602 13258 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:26.978616 11884 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20250113 08:58:26.993404 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 61459.8us
W20250113 08:58:27.048966 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 65431.3us
I20250113 08:58:27.051724 11884 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:27.060889 11963 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:58:27.027657 13348 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20250113 08:58:27.071377 13286 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:27.133482 11960 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:27.154793 12034 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:27.164309 13351 ts_tablet_manager.cc:927] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
W20250113 08:58:27.172295 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 110210us
I20250113 08:58:27.173578 13382 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250113 08:58:27.195271 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 115043us
I20250113 08:58:27.204530 12725 raft_consensus.cc:1212] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:27.230029 13351 tablet_copy_client.cc:323] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:27.217012 13376 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20250113 08:58:27.308024 12725 raft_consensus.cc:1212] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:27.380357 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250113 08:58:27.400732 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 107912us
W20250113 08:58:27.380312 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 146386us
W20250113 08:58:27.375659 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 150745us
W20250113 08:58:27.462535 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 184718us
W20250113 08:58:27.483115 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 253708us
I20250113 08:58:27.468070 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 4a9d15ff319f43f0a1dd5ae717393675 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:27.469707 13389 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.122s
I20250113 08:58:27.585500 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 4a9d15ff319f43f0a1dd5ae717393675 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-4a9d15ff319f43f0a1dd5ae717393675
W20250113 08:58:27.606277 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 136279us
W20250113 08:58:27.601821 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 150385us
W20250113 08:58:27.836813 13358 log.cc:926] Time spent T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.060s user 0.002s sys 0.000s
I20250113 08:58:27.838933 13358 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:27.840062 13358 tablet_bootstrap.cc:492] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:27.840857 13358 ts_tablet_manager.cc:1397] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.921s user 0.031s sys 0.032s
I20250113 08:58:27.846347 11884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "209260c370164f719208bd1404eef071"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:27.847103 11884 raft_consensus.cc:604] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:27.847013 11903 tablet_copy_source_session.cc:215] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
W20250113 08:58:27.866638 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 70442.6us
W20250113 08:58:27.883853 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 279688us
I20250113 08:58:27.885586 13351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a9d15ff319f43f0a1dd5ae717393675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:27.873211 13358 raft_consensus.cc:357] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:27.936126 13394 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:27.939966 13394 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:27.942009 13394 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:27.946089 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:27.938103 13358 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:27.951802 13358 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:27.954739 13358 ts_tablet_manager.cc:1428] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.114s user 0.006s sys 0.006s
I20250113 08:58:27.956802 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-9a6bba22615448a8acbbb898fb381116 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:27.957237 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-9a6bba22615448a8acbbb898fb381116 on tablet 9a6bba22615448a8acbbb898fb381116 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:27.968941 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:27.988703 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 487844us
I20250113 08:58:27.989701 13398 raft_consensus.cc:1059] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:28.024662 11931 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:28.019264 13398 consensus_queue.cc:237] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.046736 13395 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:28.050493 12723 raft_consensus.cc:1212] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:28.055500 12725 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.056941 13382 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:28.062810 11891 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.064164 13398 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:28.069773 11960 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.081624 11964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:28.097326 11964 raft_consensus.cc:604] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:28.114086 13398 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20250113 08:58:28.126210 13334 raft_consensus.cc:1059] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:28.149883 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:28.150476 11891 raft_consensus.cc:604] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:28.192209 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:28.190104 13245 raft_consensus.cc:988] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:28.193104 12037 raft_consensus.cc:604] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:28.193732 13245 raft_consensus.cc:1076] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:28.191368 13348 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.195444 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250113 08:58:28.196790 12725 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.207177 12037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "209260c370164f719208bd1404eef071"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33876
I20250113 08:58:28.207888 12037 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:28.208314 12037 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:28.208709 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:28.209417 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:28.217602 12037 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.216960 11790 catalog_manager.cc:5526] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.163177 13334 consensus_queue.cc:237] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.220999 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:28.221638 11890 raft_consensus.cc:3049] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:28.221731 12037 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:28.221971 11890 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:28.223873 11890 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.225172 11890 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:28.235512 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:28.236436 12725 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:28.237635 13351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:38504 (request call id 336) took 1090 ms. Trace:
I20250113 08:58:28.238204 13351 rpcz_store.cc:276] 0113 08:58:27.146912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:27.147825 (+ 913us) service_pool.cc:225] Handling call
0113 08:58:27.164766 (+ 16941us) ts_tablet_manager.cc:928] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
0113 08:58:27.884218 (+719452us) tablet_copy_client.cc:368] Tablet copy session begun
0113 08:58:28.096433 (+212215us) tablet_metadata.cc:641] Metadata flushed
0113 08:58:28.096804 (+ 371us) tablet_copy_client.cc:497] Wrote new tablet metadata
0113 08:58:28.190360 (+ 93556us) tablet_copy_client.cc:418] Wrote new consensus metadata
0113 08:58:28.209052 (+ 18692us) tablet_replica.cc:206] Creating consensus instance
0113 08:58:28.237464 (+ 28412us) ts_tablet_manager.cc:983] Replica registered.
0113 08:58:28.237539 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":25744,"tablet-copy.queue_time_us":7866,"thread_start_us":23207,"threads_started":1}
I20250113 08:58:28.240764 13351 tablet_copy_client.cc:806] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:28.239012 11960 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.241801 13351 tablet_copy_client.cc:670] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:28.246754 13351 tablet_copy_client.cc:538] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:28.248780 11890 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:28.250387 12006 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:28.254451 13344 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:28.255018 13344 raft_consensus.cc:695] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:28.256115 13344 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.268682 11890 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.274044 12725 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:28.316504 13351 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:28.336087 12725 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.337603 13386 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:28.342226 12037 consensus_queue.cc:237] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.351460 12034 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.357417 13386 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:28.371690 12723 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:28.375650 11787 catalog_manager.cc:5526] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:28.384770 13344 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20250113 08:58:28.416858 11964 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.418361 11960 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:28.423303 13348 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:28.430166 11960 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.431367 13351 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:28.432556 13351 tablet_bootstrap.cc:492] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:28.433950 13351 ts_tablet_manager.cc:1397] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.118s user 0.024s sys 0.026s
I20250113 08:58:28.438736 13351 raft_consensus.cc:357] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:28.439925 13351 raft_consensus.cc:738] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:28.440819 13351 consensus_queue.cc:260] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:28.451367 13381 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:28.457681 13351 ts_tablet_manager.cc:1428] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.023s user 0.009s sys 0.001s
I20250113 08:58:28.459591 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-4a9d15ff319f43f0a1dd5ae717393675 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:28.460119 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-4a9d15ff319f43f0a1dd5ae717393675 on tablet 4a9d15ff319f43f0a1dd5ae717393675 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:28.465224 13398 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.478245 12723 raft_consensus.cc:2949] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.495369 13386 raft_consensus.cc:1059] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:28.503763 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 80cc6c97986444348b1837c27c41338a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:28.604979 12725 raft_consensus.cc:1212] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:28.591012 11789 catalog_manager.cc:5526] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250113 08:58:28.674749 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 101050us
W20250113 08:58:28.702602 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 68613.8us
W20250113 08:58:28.707764 13420 log.cc:926] Time spent T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20250113 08:58:28.558130 13388 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:28.752803 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 140007us
I20250113 08:58:28.707209 13386 consensus_queue.cc:237] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:28.803264 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 192379us
I20250113 08:58:28.811753 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cc6c97986444348b1837c27c41338a with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:28.819425 13292 raft_consensus.cc:988] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:28.819994 13292 raft_consensus.cc:1076] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
W20250113 08:58:28.834111 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 114905us
W20250113 08:58:28.834714 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 157651us
I20250113 08:58:28.836189 12725 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:28.845966 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 224967us
I20250113 08:58:28.851197 12726 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.861222 11888 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:28.862082 11888 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:28.862794 11888 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:28.864023 13448 tablet_replica.cc:331] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:28.865353 13442 raft_consensus.cc:988] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:28.865854 13227 raft_consensus.cc:1076] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:28.867834 11891 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:28.872442 13448 raft_consensus.cc:2238] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:28.877970 11967 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:28.878782 11967 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:28.879232 11967 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:28.878412 11963 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.883541 11964 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:28.887676 13382 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:28.887696 13448 raft_consensus.cc:2267] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:28.889745 11787 catalog_manager.cc:5526] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:28.900219 11967 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.907074 11967 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:28.907604 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:28.908272 12725 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:28.915943 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:28.916604 11892 raft_consensus.cc:3049] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:28.916921 11892 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:28.917580 11892 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.918826 11892 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:28.920680 12725 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:28.922809 11934 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:28.929663 13441 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:28.930143 13441 raft_consensus.cc:695] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:28.930979 13441 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:28.933475 11892 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:28.900321 13434 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20250113 08:58:28.940033 13388 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:28.942652 12726 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:28.944109 13447 raft_consensus.cc:988] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:28.951442 13447 raft_consensus.cc:1076] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:28.948868 13398 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20250113 08:58:28.973768 13448 ts_tablet_manager.cc:1905] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:28.985859 11960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:28.986616 11960 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:28.987039 11960 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:28.994000 11888 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:28.995079 11884 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:29.007833 13396 consensus_queue.cc:1046] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:29.009024 13396 raft_consensus.cc:3049] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:29.009423 13396 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:29.010289 13396 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:29.011298 13396 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:29.014418 11888 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:29.033751 11891 consensus_queue.cc:237] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:29.035413 11960 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:29.040241 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:29.040994 12725 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:29.041040 11891 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:29.040241 11790 catalog_manager.cc:5526] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:29.043654 13434 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:29.048786 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:29.048955 11960 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:29.055168 11891 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:29.057031 11932 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:29.058167 12725 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.057972 13441 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:29.062073 13441 raft_consensus.cc:695] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:29.057922 13444 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.066294 13434 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.068001 11967 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.062944 13441 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:29.077543 12037 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:29.078912 13386 raft_consensus.cc:1059] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:29.078166 12034 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.087431 13386 consensus_queue.cc:237] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:29.098567 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:29.099217 12037 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:29.099598 12723 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.114342 11964 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:29.123476 11858 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:29.138381 13334 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:29.146409 13334 raft_consensus.cc:695] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:29.147655 13334 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:29.144891 12037 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:29.181651 12034 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:29.168010 11790 catalog_manager.cc:5526] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.191980 13443 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:29.206467 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:29.207130 12042 raft_consensus.cc:3049] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:29.207417 12042 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:29.208134 12042 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:29.214957 12042 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:29.222221 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:29.223790 13381 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:29.230986 13448 ts_tablet_manager.cc:1918] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:29.231509 13448 log.cc:1198] T 80cc6c97986444348b1837c27c41338a P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/80cc6c97986444348b1837c27c41338a
I20250113 08:58:29.235595 12726 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:29.235818 11964 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:29.238075 13227 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:29.238842 12042 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:29.239850 13227 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:29.243709 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 80cc6c97986444348b1837c27c41338a (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:29.261070 11788 catalog_manager.cc:5526] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:29.264858 12039 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.277164 11787 catalog_manager.cc:5526] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:29.278119 13444 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.280020 11967 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.294750 11891 consensus_queue.cc:237] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:29.302791 12726 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:29.302187 11788 catalog_manager.cc:5526] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250113 08:58:29.305219 12005 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.10.115.190:41823, user_credentials={real_user=slave}} blocked reactor thread for 97197.5us
I20250113 08:58:29.305049 13227 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:29.319962 13437 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.321472 11967 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:29.326157 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3345ca9b1640a4b4e6d2efd2ee061d with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:29.326277 11787 catalog_manager.cc:5526] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.329010 13386 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250113 08:58:29.346926 12037 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:29.347903 12039 raft_consensus.cc:2949] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.349808 11967 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.316728 12725 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.359280 12726 raft_consensus.cc:2949] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.361959 13464 tablet_replica.cc:331] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:29.363368 13464 raft_consensus.cc:2238] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:29.368649 13227 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:29.396651 13464 raft_consensus.cc:2267] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:29.378727 11787 catalog_manager.cc:5526] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.384516 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9a6bba22615448a8acbbb898fb381116 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:29.403822 13464 ts_tablet_manager.cc:1905] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:29.386682 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:29.427068 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6bba22615448a8acbbb898fb381116 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:29.487146 12726 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:29.494076 13375 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20250113 08:58:29.509020 12033 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:29.507386 11891 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:29.510885 11892 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:29.526208 13395 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:29.528864 13439 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:29.530184 13439 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:29.518287 13465 tablet_replica.cc:331] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:29.561592 12041 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:29.561110 13465 raft_consensus.cc:2238] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:29.570039 13465 raft_consensus.cc:2267] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:29.577612 13464 ts_tablet_manager.cc:1918] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:29.578171 13464 log.cc:1198] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/2c3345ca9b1640a4b4e6d2efd2ee061d
I20250113 08:58:29.579928 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 2c3345ca9b1640a4b4e6d2efd2ee061d (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:29.613739 11967 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:29.610800 13465 ts_tablet_manager.cc:1905] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:29.616788 13334 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:29.679927 13465 ts_tablet_manager.cc:1918] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:29.680402 13465 log.cc:1198] T 9a6bba22615448a8acbbb898fb381116 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/9a6bba22615448a8acbbb898fb381116
I20250113 08:58:29.682690 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 9a6bba22615448a8acbbb898fb381116 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:29.685894 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:29.686844 11891 raft_consensus.cc:604] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:29.691299 12041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:29.691882 12041 raft_consensus.cc:604] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:29.694669 11964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:29.695201 11964 raft_consensus.cc:604] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:29.702872 12041 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:29.706590 13334 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:29.707623 11967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:29.709340 11967 raft_consensus.cc:604] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:29.710475 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250113 08:58:29.712113 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:29.712740 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250113 08:58:29.967464 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 53527us
I20250113 08:58:29.977986 13474 raft_consensus.cc:988] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:29.978441 13474 raft_consensus.cc:1076] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:29.981719 11964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:29.982404 11964 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:29.982853 11964 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:29.991954 13437 raft_consensus.cc:988] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:29.992465 13437 raft_consensus.cc:1076] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:29.994189 11964 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:29.996013 11964 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:29.998157 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:29.998871 12041 raft_consensus.cc:3049] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:29.999269 12041 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:30.000600 12041 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:30.002599 12041 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.006932 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:30.014170 12041 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:30.007591 11892 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.015192 11931 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:30.016074 13395 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:30.016618 13395 raft_consensus.cc:695] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:30.017601 13395 consensus_queue.cc:237] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:30.026199 11892 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:30.038972 12037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33876
I20250113 08:58:30.039695 12037 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:30.040056 12037 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.046711 13392 raft_consensus.cc:988] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:30.047217 13392 raft_consensus.cc:1076] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:30.047665 12037 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.048965 11888 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:30.049535 11888 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:30.049894 11888 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.050793 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:30.051110 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:30.051391 11890 raft_consensus.cc:3049] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:30.051702 12725 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.051728 11890 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:30.057816 11890 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.058290 11888 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.059042 11890 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.062623 11892 consensus_queue.cc:237] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.067843 12725 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:30.074419 11888 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:30.074487 12726 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:30.076243 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:30.082588 12008 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:30.082664 12723 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.083588 12037 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:30.083664 13374 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:30.084546 13374 raft_consensus.cc:695] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:30.085196 11890 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:30.085530 13374 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.090816 12723 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:30.078184 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:30.093308 11964 raft_consensus.cc:3049] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:30.093662 11964 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:30.094638 11964 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.097218 13386 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:30.103665 11964 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.120185 12037 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:30.121750 11859 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:30.122731 13227 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:30.123175 13227 raft_consensus.cc:695] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:30.123982 13227 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.138909 13386 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:30.140019 13334 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:30.140987 11787 catalog_manager.cc:5526] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:30.147305 12041 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:30.150475 12723 raft_consensus.cc:2949] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:30.164667 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 4a9d15ff319f43f0a1dd5ae717393675 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:30.179919 13390 raft_consensus.cc:988] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:30.180464 13390 raft_consensus.cc:1076] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:30.175307 11964 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:30.186734 12723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38492
I20250113 08:58:30.187271 12723 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:30.188114 12723 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:30.189026 11790 catalog_manager.cc:5526] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:30.195621 12723 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.198632 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:30.198922 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:30.199239 11964 raft_consensus.cc:3049] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:30.199532 11888 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:30.199553 11964 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:30.200383 11964 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.201468 11964 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:30.201033 11788 catalog_manager.cc:5526] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:30.205210 11789 catalog_manager.cc:5526] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:30.207669 11888 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:30.208405 11964 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:30.209288 12723 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:30.209599 12686 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:30.210381 13406 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:30.210847 13406 raft_consensus.cc:695] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:30.211649 13406 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.225965 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a9d15ff319f43f0a1dd5ae717393675 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:30.229836 13483 tablet_replica.cc:331] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:30.231014 13483 raft_consensus.cc:2238] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:30.231801 13483 raft_consensus.cc:2267] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:30.236502 11788 catalog_manager.cc:5526] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 3 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:30.237578 13483 ts_tablet_manager.cc:1905] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:30.257615 13483 ts_tablet_manager.cc:1918] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:30.257937 13483 log.cc:1198] T 4a9d15ff319f43f0a1dd5ae717393675 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/4a9d15ff319f43f0a1dd5ae717393675
I20250113 08:58:30.259385 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 4a9d15ff319f43f0a1dd5ae717393675 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:30.499070 11890 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:30.500260 13395 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:30.515451 12041 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:30.517760 13390 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:30.521765 11888 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:30.529759 13432 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:30.550462 12723 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:30.555714 13386 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:30.560248 11967 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:30.564240 12725 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:30.564875 13334 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:30.565937 13374 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250113 08:58:30.633075 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 73018.6us
I20250113 08:58:30.687026 11890 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:30.688256 13289 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:30.698683 11964 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:30.704260 13406 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:30.862141 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:30.909272 11964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:30.909874 11964 raft_consensus.cc:604] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:30.912423 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:30.912887 12037 raft_consensus.cc:604] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:30.915287 11888 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:30.915864 11888 raft_consensus.cc:604] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:30.918313 11964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:30.918725 11964 raft_consensus.cc:604] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:30.919554 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250113 08:58:30.920816 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:30.921402 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:30.975036 13390 raft_consensus.cc:988] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:30.975750 13441 raft_consensus.cc:1076] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:30.979620 12725 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38492
I20250113 08:58:30.981006 12725 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:30.981504 12725 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.990550 12725 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.992556 12725 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:30.994609 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:30.995471 11964 raft_consensus.cc:3049] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:30.995781 11964 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:30.996538 11964 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:30.998713 11964 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:30.993770 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:31.019454 11888 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:31.026196 11964 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:31.027181 11888 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:31.027941 12686 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:31.028986 13406 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:31.029429 13406 raft_consensus.cc:695] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:31.030282 13406 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:31.049996 11789 catalog_manager.cc:5526] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:31.110500 13334 raft_consensus.cc:988] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:31.111119 13437 raft_consensus.cc:1076] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:31.112610 12037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33876
I20250113 08:58:31.113155 12037 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:31.113582 12037 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:31.119666 12037 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:31.121026 12037 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:31.122087 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:31.122727 11964 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:31.122982 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:31.123571 11888 raft_consensus.cc:3049] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:31.123898 11888 raft_consensus.cc:738] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:31.124790 11888 consensus_queue.cc:260] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:31.126163 11888 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:31.130620 11964 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:31.132511 12007 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:31.133339 13432 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:31.133857 13432 raft_consensus.cc:695] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:31.134753 13432 consensus_queue.cc:237] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:31.136476 11888 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:31.151747 11789 catalog_manager.cc:5526] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:31.249368 11964 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:31.255463 12033 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:31.255502 11934 consensus_peers.cc:487] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e1a8409abe745918b8f11dc8590376a. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:31.256801 13390 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:31.257218 11888 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:31.258486 13395 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:31.264456 13390 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.266319 12033 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.282871 11964 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:31.290344 11789 catalog_manager.cc:5526] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:31.283074 11888 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.327481 12033 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:31.329800 13441 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:31.334095 11888 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:31.335995 13392 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250113 08:58:31.342677 11934 consensus_peers.cc:487] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8133c246c9d74113aec7a002df391ef7. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:31.342499 13441 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.344538 12033 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.366644 11788 catalog_manager.cc:5526] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.381147 13374 raft_consensus.cc:988] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:31.381680 13374 raft_consensus.cc:1076] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:31.383616 11964 consensus_queue.cc:237] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:31.386229 11962 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:31.386782 11962 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:31.387140 11962 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:31.397027 11962 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:31.398326 11890 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.399847 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:31.400463 12725 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:31.405767 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:31.406507 12033 raft_consensus.cc:3049] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:31.406828 12033 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:31.407131 12725 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:31.409713 11934 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:31.410537 13441 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:31.410984 13441 raft_consensus.cc:695] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:31.411828 13441 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:31.407857 11962 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:31.407905 12033 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:31.417801 12033 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:31.432176 13390 raft_consensus.cc:988] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:31.432721 13390 raft_consensus.cc:1076] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:31.434554 11888 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:31.435156 11888 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:31.435479 11888 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:31.444370 11888 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:31.446183 12033 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:31.450124 11888 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:31.450631 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:31.451236 12041 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:58:31.453305 11934 consensus_peers.cc:487] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 099cf849e7414101a8741339c72fa73e. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:31.453678 12037 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:31.455099 13395 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:31.455888 11888 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:31.457274 13392 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:31.460346 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:31.461130 11962 raft_consensus.cc:3049] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:31.461879 11962 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:31.462960 11962 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:31.465466 11962 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:31.474493 12041 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:31.478969 11856 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:31.480209 13437 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:31.480847 13437 raft_consensus.cc:695] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:31.481712 13437 consensus_queue.cc:237] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:31.490523 13392 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.501637 11890 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:31.504598 13491 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:31.507079 11788 catalog_manager.cc:5526] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:31.508500 11962 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:31.519363 11967 raft_consensus.cc:1270] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:31.522198 13374 consensus_queue.cc:1035] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:31.534847 11962 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:31.536842 13491 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:31.534605 12033 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.663972 11789 catalog_manager.cc:5526] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:31.686118 11788 catalog_manager.cc:5526] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:31.660341 12042 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:31.705060 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 103254us
I20250113 08:58:31.728322 11890 raft_consensus.cc:1270] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250113 08:58:31.749974 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 258930us
I20250113 08:58:31.794004 13528 consensus_queue.cc:1035] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250113 08:58:31.834844 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 83792.6us
I20250113 08:58:31.853927 11962 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250113 08:58:31.854061 12008 consensus_peers.cc:487] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 531b33203b294592b3a1275dc74ab4f3. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:31.856911 13432 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20250113 08:58:31.891495 13545 ts_tablet_manager.cc:927] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:31.892292 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 139801us
I20250113 08:58:31.900568 12042 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:31.902977 13547 ts_tablet_manager.cc:927] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:31.906581 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 139090us
I20250113 08:58:31.932576 13547 tablet_copy_client.cc:323] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:31.949036 11890 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.947691 13545 tablet_copy_client.cc:323] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:31.960135 11888 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:31.962242 13432 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:31.980612 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250113 08:58:32.011910 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 47118.1us
W20250113 08:58:32.013764 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 54888.3us
W20250113 08:58:32.031219 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 78028.2us
I20250113 08:58:32.025396 13536 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250113 08:58:32.069792 13560 ts_tablet_manager.cc:927] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:32.071897 12033 raft_consensus.cc:1270] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250113 08:58:32.076885 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 153504us
I20250113 08:58:32.082788 13553 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:32.093801 12722 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:32.096072 13395 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250113 08:58:32.117643 11892 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:32.123333 11962 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:32.123970 11962 raft_consensus.cc:604] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:32.131304 13551 raft_consensus.cc:988] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:32.132264 13534 raft_consensus.cc:1076] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:32.132587 13395 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:32.148034 11891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:32.148598 11891 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:32.148943 11891 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:32.159348 11888 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:32.159986 11888 raft_consensus.cc:604] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:32.155494 11787 catalog_manager.cc:5526] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:32.165521 11978 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 3e1a8409abe745918b8f11dc8590376a from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:32.165683 11977 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 099cf849e7414101a8741339c72fa73e from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:32.166190 11978 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 3e1a8409abe745918b8f11dc8590376a from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-3e1a8409abe745918b8f11dc8590376a
I20250113 08:58:32.167645 11977 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 099cf849e7414101a8741339c72fa73e from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-099cf849e7414101a8741339c72fa73e
I20250113 08:58:32.188642 11978 tablet_copy_source_session.cc:215] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:32.195915 12039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:32.196535 12039 raft_consensus.cc:604] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:32.197913 11891 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:32.204113 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250113 08:58:32.206101 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:32.206800 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:32.212247 13560 tablet_copy_client.cc:323] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:32.213088 11891 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:32.218118 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:32.218956 12039 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:32.221763 13547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1a8409abe745918b8f11dc8590376a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:32.226145 11964 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:32.230515 13537 raft_consensus.cc:988] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:32.231783 11962 raft_consensus.cc:1270] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:32.232239 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:32.233759 13527 raft_consensus.cc:1076] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:32.233772 11963 raft_consensus.cc:3049] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:32.234184 11963 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:32.234908 11963 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:32.235334 11960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:32.235870 11960 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:32.236116 11963 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:32.236259 11960 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:32.238384 11978 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet 8133c246c9d74113aec7a002df391ef7 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:32.238914 11978 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet 8133c246c9d74113aec7a002df391ef7 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-8133c246c9d74113aec7a002df391ef7
I20250113 08:58:32.284199 12039 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:32.286000 11960 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:32.287993 11960 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:32.288969 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:32.289708 12039 raft_consensus.cc:3049] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:32.290066 12039 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:32.285624 11856 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:32.292934 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:32.293529 11892 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:32.290987 12039 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:32.298072 12039 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:32.309422 11963 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:32.311522 13556 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:32.313961 13334 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:32.314568 13334 raft_consensus.cc:695] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:32.323268 11892 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:32.329219 11932 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:32.315515 11977 tablet_copy_source_session.cc:215] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:32.352862 12039 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:32.333470 13392 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:32.315500 13334 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:32.370297 11978 tablet_copy_source_session.cc:215] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:32.375777 13392 raft_consensus.cc:695] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:32.409783 13392 consensus_queue.cc:237] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:32.459455 13572 ts_tablet_manager.cc:927] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:32.462414 13545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099cf849e7414101a8741339c72fa73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:32.467859 13560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8133c246c9d74113aec7a002df391ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:32.733888 13547 tablet_copy_client.cc:806] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:32.734637 13547 tablet_copy_client.cc:670] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
W20250113 08:58:32.743577 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 178101us
I20250113 08:58:32.735575 13572 tablet_copy_client.cc:323] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:32.750286 11787 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:32.787671 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 531b33203b294592b3a1275dc74ab4f3 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:32.788517 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 531b33203b294592b3a1275dc74ab4f3 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-531b33203b294592b3a1275dc74ab4f3
I20250113 08:58:32.899416 13553 raft_consensus.cc:988] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:32.900017 13553 raft_consensus.cc:1076] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:32.909884 11891 raft_consensus.cc:1270] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:32.919592 12042 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:32.934006 13584 consensus_queue.cc:1035] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:32.937273 13547 tablet_copy_client.cc:538] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:32.944909 13562 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20250113 08:58:32.951854 11788 catalog_manager.cc:5526] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 2 to 3, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:33.002605 11964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:33.004734 11964 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:33.005223 11964 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:58:32.997754 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 78733.2us
W20250113 08:58:33.030848 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 48091.6us
I20250113 08:58:32.965178 12052 tablet_copy_source_session.cc:215] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:33.037014 11964 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:33.068763 11964 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:33.073737 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:33.055310 13547 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:33.074956 11891 raft_consensus.cc:3049] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:33.075452 11891 raft_consensus.cc:738] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:33.075610 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:33.076347 12726 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:33.076601 11891 consensus_queue.cc:260] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:33.078480 11891 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:33.091778 11963 raft_consensus.cc:1235] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:33.096365 11891 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:33.097640 11932 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:33.103328 13560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:38492 (request call id 224) took 1175 ms. Trace:
I20250113 08:58:33.104182 13572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531b33203b294592b3a1275dc74ab4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:33.103823 13560 rpcz_store.cc:276] 0113 08:58:31.928075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:31.928218 (+ 143us) service_pool.cc:225] Handling call
0113 08:58:32.070732 (+142514us) ts_tablet_manager.cc:928] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
0113 08:58:32.464496 (+393764us) tablet_copy_client.cc:368] Tablet copy session begun
0113 08:58:32.893928 (+429432us) tablet_metadata.cc:641] Metadata flushed
0113 08:58:32.895996 (+ 2068us) tablet_copy_client.cc:497] Wrote new tablet metadata
0113 08:58:33.102583 (+206587us) tablet_copy_client.cc:418] Wrote new consensus metadata
0113 08:58:33.102877 (+ 294us) tablet_replica.cc:206] Creating consensus instance
0113 08:58:33.103206 (+ 329us) ts_tablet_manager.cc:983] Replica registered.
0113 08:58:33.103260 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":2084,"tablet-copy.queue_time_us":141059,"thread_start_us":305178,"threads_started":2}
I20250113 08:58:33.111790 13560 tablet_copy_client.cc:806] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:33.112483 13560 tablet_copy_client.cc:670] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:33.110955 12726 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:33.122802 13395 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:33.126888 13395 raft_consensus.cc:695] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:33.130342 13545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:38492 (request call id 222) took 1281 ms. Trace:
I20250113 08:58:33.130937 13545 rpcz_store.cc:276] 0113 08:58:31.848619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:31.848770 (+ 151us) service_pool.cc:225] Handling call
0113 08:58:31.913673 (+ 64903us) ts_tablet_manager.cc:928] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
0113 08:58:32.412961 (+499288us) tablet_copy_client.cc:368] Tablet copy session begun
0113 08:58:32.767456 (+354495us) tablet_metadata.cc:641] Metadata flushed
0113 08:58:32.767816 (+ 360us) tablet_copy_client.cc:497] Wrote new tablet metadata
0113 08:58:33.121576 (+353760us) tablet_copy_client.cc:418] Wrote new consensus metadata
0113 08:58:33.129746 (+ 8170us) tablet_replica.cc:206] Creating consensus instance
0113 08:58:33.130140 (+ 394us) ts_tablet_manager.cc:983] Replica registered.
0113 08:58:33.130261 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":225,"tablet-copy.queue_time_us":42088,"thread_start_us":62012,"threads_started":2}
I20250113 08:58:33.128232 13395 consensus_queue.cc:237] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:33.138659 13545 tablet_copy_client.cc:806] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:33.141228 13545 tablet_copy_client.cc:670] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:33.163436 11961 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:33.173713 12033 raft_consensus.cc:1270] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:33.184437 13562 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:33.189798 13395 consensus_queue.cc:1035] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20250113 08:58:33.225580 13545 tablet_copy_client.cc:538] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:33.228840 13560 tablet_copy_client.cc:538] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:33.263008 13560 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:33.271299 13572 tablet_copy_client.cc:806] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:33.272054 13572 tablet_copy_client.cc:670] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:33.274806 13545 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:33.279778 13572 tablet_copy_client.cc:538] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:33.291414 13547 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:33.292793 13547 tablet_bootstrap.cc:492] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:33.293808 13547 ts_tablet_manager.cc:1397] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.239s user 0.018s sys 0.019s
I20250113 08:58:33.404801 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:33.420578 13547 raft_consensus.cc:357] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.444291 13547 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:33.479022 13547 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.482050 13572 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:33.526798 13547 ts_tablet_manager.cc:1428] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.232s user 0.005s sys 0.006s
I20250113 08:58:33.529646 11977 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-3e1a8409abe745918b8f11dc8590376a received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:33.533372 11977 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-3e1a8409abe745918b8f11dc8590376a on tablet 3e1a8409abe745918b8f11dc8590376a with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:33.542196 13545 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:33.544042 13545 tablet_bootstrap.cc:492] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:33.556684 13560 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:33.559819 12725 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250113 08:58:33.563139 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 73481.6us
I20250113 08:58:33.605495 13546 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:33.607535 13545 ts_tablet_manager.cc:1397] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.333s user 0.031s sys 0.031s
I20250113 08:58:33.619488 13560 tablet_bootstrap.cc:492] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:33.650362 13545 raft_consensus.cc:357] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.651683 13545 raft_consensus.cc:738] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:33.653275 13560 ts_tablet_manager.cc:1397] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.390s user 0.026s sys 0.013s
I20250113 08:58:33.660732 11787 catalog_manager.cc:5526] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:33.659091 13545 consensus_queue.cc:260] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.674186 11891 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:33.676805 13390 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:33.680878 13545 ts_tablet_manager.cc:1428] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.072s user 0.004s sys 0.005s
I20250113 08:58:33.686163 11977 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-099cf849e7414101a8741339c72fa73e received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:33.686757 11977 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-099cf849e7414101a8741339c72fa73e on tablet 099cf849e7414101a8741339c72fa73e with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:33.709291 12722 raft_consensus.cc:1212] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:33.687512 13560 raft_consensus.cc:357] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.728277 13560 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:33.741976 13560 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.750340 13560 ts_tablet_manager.cc:1428] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.097s user 0.009s sys 0.000s
I20250113 08:58:33.752375 11977 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-8133c246c9d74113aec7a002df391ef7 received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:33.752875 11977 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-8133c246c9d74113aec7a002df391ef7 on tablet 8133c246c9d74113aec7a002df391ef7 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:33.758193 13572 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:33.759752 13572 tablet_bootstrap.cc:492] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:33.760481 13572 ts_tablet_manager.cc:1397] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.279s user 0.018s sys 0.021s
I20250113 08:58:33.763679 13572 raft_consensus.cc:357] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.764709 13572 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:33.765446 13572 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:33.769289 13572 ts_tablet_manager.cc:1428] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20250113 08:58:33.771230 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-531b33203b294592b3a1275dc74ab4f3 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:33.771764 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-531b33203b294592b3a1275dc74ab4f3 on tablet 531b33203b294592b3a1275dc74ab4f3 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:33.807525 12722 raft_consensus.cc:1212] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250113 08:58:33.842890 11961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:33.843683 11961 raft_consensus.cc:604] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:33.848053 12042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "992519c5663649019a1ff9753075ddc7"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:33.848644 12042 raft_consensus.cc:604] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:33.851533 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:33.852109 11891 raft_consensus.cc:604] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:33.854427 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250113 08:58:33.855774 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:33.856361 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:33.881415 13584 raft_consensus.cc:988] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:33.882020 13584 raft_consensus.cc:1076] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:33.884518 12033 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33878
I20250113 08:58:33.885228 12033 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:33.885720 12033 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:33.901829 12033 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:33.909479 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:33.910805 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:33.910969 11891 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:33.911563 11961 raft_consensus.cc:3049] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:33.912406 11961 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:33.910367 12033 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:33.915808 11961 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:33.926152 11891 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:33.932835 11961 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:33.937173 12006 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:33.939857 13432 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:33.941045 13432 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:33.942566 13432 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:33.952576 11961 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:34.008644 11788 catalog_manager.cc:5526] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 3 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:34.024132 12722 raft_consensus.cc:1212] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:34.026510 12726 raft_consensus.cc:1212] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:34.079182 13592 raft_consensus.cc:988] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:34.079710 13592 raft_consensus.cc:1076] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:34.102975 12035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33876
I20250113 08:58:34.103624 12035 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:34.104018 12035 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:34.153587 13542 raft_consensus.cc:988] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:34.154193 13432 raft_consensus.cc:1076] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:34.161386 11963 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "992519c5663649019a1ff9753075ddc7"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:34.163483 11963 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:34.164021 11963 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:34.167275 12035 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.176705 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:34.177450 11890 raft_consensus.cc:3049] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:34.177736 11890 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:34.178577 11890 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.179631 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:34.180320 12726 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:34.180459 11890 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:34.183151 12035 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:34.176587 11963 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:34.191267 11963 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:34.191478 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:34.192139 11891 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:34.192487 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:34.193078 12033 raft_consensus.cc:3049] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:34.193378 12033 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:34.194149 12033 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:34.195399 12033 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:34.199353 13594 raft_consensus.cc:1059] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:34.206611 11890 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:34.203505 13594 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.210018 12006 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:34.211709 11891 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:34.212808 11932 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:34.216620 13541 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:34.216938 13546 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:34.217445 13546 raft_consensus.cc:695] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:34.218230 13546 consensus_queue.cc:237] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:34.217423 13541 raft_consensus.cc:695] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:34.222116 13541 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.222877 12033 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:34.230687 12726 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:34.248800 11788 catalog_manager.cc:5526] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 2 to 3, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:34.264770 13584 raft_consensus.cc:1059] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:34.267443 13584 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.297031 12726 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.298766 11790 catalog_manager.cc:5526] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:34.299856 12035 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.306563 13546 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.306913 11891 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.311895 13594 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.327872 13621 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:34.335177 12722 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:34.335389 11884 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:34.338968 12033 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:34.339359 13546 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:34.340477 13594 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:34.351075 13621 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:34.369824 13594 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.371275 12033 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.387738 11891 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.396847 11790 catalog_manager.cc:5526] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:34.408267 13621 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.404682 12726 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.436374 12722 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.434497 11960 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.445932 12035 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.453398 11884 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:34.457145 13390 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:34.409930 11891 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.462134 11788 catalog_manager.cc:5526] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:34.469555 13546 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.471681 11884 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.493614 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 3e1a8409abe745918b8f11dc8590376a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:34.509146 12724 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:34.510506 11963 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.512444 13620 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:34.518038 12724 raft_consensus.cc:2949] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.523433 11891 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:34.536159 13541 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.544286 11788 catalog_manager.cc:5526] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:34.564939 13620 raft_consensus.cc:1059] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:34.568044 13620 consensus_queue.cc:237] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:34.577332 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1a8409abe745918b8f11dc8590376a with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:34.589991 11890 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.591068 12033 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.591784 13546 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.591938 12726 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.597676 13594 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.600096 13584 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.603893 13634 tablet_replica.cc:331] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:34.613318 12722 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:34.614923 13584 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:34.617511 13634 raft_consensus.cc:2238] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:34.618459 13634 raft_consensus.cc:2267] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:34.622758 13621 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.624619 12722 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.626272 13634 ts_tablet_manager.cc:1905] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:34.636871 12037 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:34.651520 13620 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:34.659482 12724 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:34.659193 12035 raft_consensus.cc:2949] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.661922 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8133c246c9d74113aec7a002df391ef7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:34.661383 13631 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250113 08:58:34.662727 11790 catalog_manager.cc:5526] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 4 to 5, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.673377 11963 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:34.670095 13630 raft_consensus.cc:1059] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:34.696889 13630 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:34.748394 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 62641.7us
I20250113 08:58:34.754222 13640 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.755055 11892 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:34.757308 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 8133c246c9d74113aec7a002df391ef7 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52584
W20250113 08:58:34.769227 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 58435.3us
I20250113 08:58:34.775981 11888 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:34.775982 13594 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:34.782615 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 71145.1us
I20250113 08:58:34.779552 12033 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.802659 13634 ts_tablet_manager.cc:1918] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:34.803228 13634 log.cc:1198] T 3e1a8409abe745918b8f11dc8590376a P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/3e1a8409abe745918b8f11dc8590376a
I20250113 08:58:34.806581 12725 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.805028 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 3e1a8409abe745918b8f11dc8590376a (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:34.823128 13631 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:34.794924 13621 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20250113 08:58:34.848979 11960 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250113 08:58:34.850185 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 47464.2us
I20250113 08:58:34.856865 12037 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:34.865880 11884 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:34.859634 13542 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20250113 08:58:34.868260 13646 tablet_replica.cc:331] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:34.868881 13645 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20250113 08:58:34.869416 13584 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250113 08:58:34.872483 13647 log.cc:926] Time spent T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20250113 08:58:34.904975 11884 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:34.917827 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:34.923290 11790 catalog_manager.cc:5526] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.005898 12726 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:35.067871 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 94753.8us
I20250113 08:58:35.013430 13639 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.137s
I20250113 08:58:35.071398 13646 raft_consensus.cc:2238] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Raft consensus shutting down.
W20250113 08:58:35.081547 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 114845us
W20250113 08:58:35.088313 13652 log.cc:926] Time spent T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.104s user 0.002s sys 0.000s
I20250113 08:58:35.094000 13646 raft_consensus.cc:2267] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:35.137969 13646 ts_tablet_manager.cc:1905] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250113 08:58:35.149441 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 88397.6us
I20250113 08:58:35.160414 13646 ts_tablet_manager.cc:1918] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250113 08:58:35.160948 13646 log.cc:1198] T 8133c246c9d74113aec7a002df391ef7 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/8133c246c9d74113aec7a002df391ef7
I20250113 08:58:35.162768 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 8133c246c9d74113aec7a002df391ef7 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:35.170902 13643 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.186484 11963 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.205896 12725 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:35.222208 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 87792.4us
I20250113 08:58:35.275393 11790 catalog_manager.cc:5526] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.295050 12037 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.300493 12725 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:35.301952 13643 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:35.308760 13642 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.312501 11963 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:35.314746 13640 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:35.318955 11962 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:35.331763 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 65237.9us
I20250113 08:58:35.332293 12725 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.343050 11961 consensus_queue.cc:237] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.345238 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 531b33203b294592b3a1275dc74ab4f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:35.352399 12722 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:35.354092 13594 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:35.361660 11788 catalog_manager.cc:5526] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:35.362655 13663 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.370673 12725 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.374951 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 531b33203b294592b3a1275dc74ab4f3 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:35.400987 11890 raft_consensus.cc:2949] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.408358 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 099cf849e7414101a8741339c72fa73e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:35.420831 13646 tablet_replica.cc:331] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:35.423144 12033 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:35.423826 12033 raft_consensus.cc:604] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:35.429159 11890 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:35.429811 13646 raft_consensus.cc:2238] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:35.441116 11790 catalog_manager.cc:5526] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.442209 13646 raft_consensus.cc:2267] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:35.437299 12035 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:35.443976 12035 raft_consensus.cc:604] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:35.439031 13657 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:35.446388 13646 ts_tablet_manager.cc:1905] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:35.450289 11961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:35.450951 11961 raft_consensus.cc:604] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:35.453660 11961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:35.454274 11961 raft_consensus.cc:604] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:35.451175 11890 raft_consensus.cc:2949] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:35.456279 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250113 08:58:35.458190 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:35.458868 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:35.464651 13646 ts_tablet_manager.cc:1918] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:35.465276 13646 log.cc:1198] T 531b33203b294592b3a1275dc74ab4f3 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/531b33203b294592b3a1275dc74ab4f3
I20250113 08:58:35.467080 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 531b33203b294592b3a1275dc74ab4f3 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:35.472102 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 099cf849e7414101a8741339c72fa73e with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:35.474942 13672 tablet_replica.cc:331] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:35.476104 13672 raft_consensus.cc:2238] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:35.481421 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 531b33203b294592b3a1275dc74ab4f3 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:35.482355 13672 raft_consensus.cc:2267] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:35.486588 13672 ts_tablet_manager.cc:1905] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:35.494016 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 531b33203b294592b3a1275dc74ab4f3 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:35.509649 13672 ts_tablet_manager.cc:1918] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:35.510231 13672 log.cc:1198] T 099cf849e7414101a8741339c72fa73e P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/099cf849e7414101a8741339c72fa73e
I20250113 08:58:35.512012 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 099cf849e7414101a8741339c72fa73e (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:35.615486 13640 raft_consensus.cc:988] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:35.616122 13642 raft_consensus.cc:1076] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:35.617491 12726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38520
I20250113 08:58:35.617991 12726 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:35.618294 12726 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:35.624289 12726 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.626336 12726 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:35.627328 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:35.627960 12035 raft_consensus.cc:3049] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:35.628319 12035 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:35.628193 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:35.628873 11890 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:35.629066 12035 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.630267 12035 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:35.640048 12035 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:35.644564 11890 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:35.644965 12684 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:35.647110 13581 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:35.647626 13581 raft_consensus.cc:695] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:35.648562 13581 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.682394 11789 catalog_manager.cc:5526] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 3, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 3 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:35.712126 13657 raft_consensus.cc:988] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:35.712577 13594 raft_consensus.cc:1076] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:35.714150 11890 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:35.714697 11890 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:35.714977 11890 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.720441 11890 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:35.722066 11890 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:35.723402 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:35.723775 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:35.724009 12033 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.724378 11961 raft_consensus.cc:3049] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:35.724689 11961 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:35.729146 11961 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:35.730381 11961 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.731019 12033 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:35.732254 11856 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:35.733114 13516 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:35.733598 13516 raft_consensus.cc:695] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:35.734412 13516 consensus_queue.cc:237] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:35.738353 11961 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:35.749693 11789 catalog_manager.cc:5526] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:35.758065 13657 raft_consensus.cc:988] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:35.758594 13661 raft_consensus.cc:1076] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:35.762370 11884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:35.762970 11884 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:35.763306 11884 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.776701 11884 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.779517 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:35.780076 11963 raft_consensus.cc:3049] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:35.780362 11963 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:35.781044 11963 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.784385 11963 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.786767 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:35.787420 12726 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.782756 11884 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:35.794711 11963 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:35.795812 11858 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:35.798108 13516 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:35.806386 13516 raft_consensus.cc:695] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:35.811677 12726 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:35.812453 13516 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:35.836087 11789 catalog_manager.cc:5526] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 1 to 2, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:35.843886 13643 raft_consensus.cc:988] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:35.844395 13643 raft_consensus.cc:1076] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:35.846155 11961 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:35.846729 11961 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:35.847061 11961 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.853410 11961 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:35.855094 11961 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:35.855504 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:35.855664 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:35.856241 11890 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.856302 12035 raft_consensus.cc:3049] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:35.856719 12035 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:35.857471 12035 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:35.858722 12035 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:35.864166 11890 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:35.865546 11932 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:35.866437 13594 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:35.866937 13594 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:35.867749 13594 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:35.869225 12035 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:35.888100 11789 catalog_manager.cc:5526] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:36.080783 11884 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:36.082032 13662 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:36.091756 12035 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:36.093652 13664 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:36.175916 11961 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:36.177042 13641 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.191898 12035 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:36.196271 13660 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.201550 11961 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:36.203603 13660 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:36.221691 12726 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:36.227340 13516 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:36.319469 12035 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:36.320593 13594 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.330238 11884 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:36.331997 13594 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.478442 12035 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:36.485100 11963 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:36.485100 11890 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250113 08:58:36.486618 12008 consensus_peers.cc:487] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a1ace8d6d3d488583610962dabdee94. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:36.486984 13640 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:36.488005 13690 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:36.497752 13640 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.515775 11890 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.516638 11961 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.523540 11884 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:36.532541 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:36.530237 11789 catalog_manager.cc:5526] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:36.544138 12035 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:36.548578 13660 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:36.551290 11961 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250113 08:58:36.552145 11859 consensus_peers.cc:487] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 043006a99fa54df3a6ee86ebd18003a7. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:36.553791 13660 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:36.568524 13660 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.573858 12037 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.584467 11961 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.600407 11890 consensus_queue.cc:237] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:36.605818 11788 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250113 08:58:36.626533 11859 consensus_peers.cc:487] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39676b99e4434f11b6017bb867fb7471. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:36.631562 11963 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:36.632551 12037 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:36.634071 13641 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.635078 13660 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.640583 11961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:36.641273 11961 raft_consensus.cc:604] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:36.643965 11890 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:36.644496 11890 raft_consensus.cc:604] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:36.644340 13516 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.646214 11963 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.647467 11890 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:36.647986 11890 raft_consensus.cc:604] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:36.649122 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250113 08:58:36.650833 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:36.651429 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:36.647918 12037 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.658694 13660 raft_consensus.cc:988] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:36.659154 13660 raft_consensus.cc:1076] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:36.660676 11962 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:36.661175 11962 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:36.661571 11962 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:36.667287 11963 consensus_queue.cc:237] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:36.667361 11788 catalog_manager.cc:5526] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.677723 11962 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:36.688597 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:36.689173 11890 raft_consensus.cc:3049] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:36.689491 11890 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:36.690136 11890 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:36.691160 11890 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:36.697897 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:36.698596 12035 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:36.705760 12035 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:36.707018 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:36.708030 13594 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:36.708576 13594 raft_consensus.cc:695] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:36.703178 11890 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:36.709506 13594 consensus_queue.cc:237] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:36.713366 11962 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:36.722241 11790 catalog_manager.cc:5526] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 2 to 3, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
W20250113 08:58:36.727319 11934 consensus_peers.cc:487] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6a40dc5e61345558c943df4be928468. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:36.734843 12037 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:36.736304 13657 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.746201 11884 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:36.748188 13657 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:36.747426 13594 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.749979 12035 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.754148 11884 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.774152 11790 catalog_manager.cc:5526] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:36.799537 13706 raft_consensus.cc:988] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:36.800668 13709 raft_consensus.cc:1076] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:36.802224 12035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33878
I20250113 08:58:36.802752 12035 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:36.803093 12035 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:36.812350 12035 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:36.814136 12035 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:36.814771 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:36.815119 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:36.815693 11963 raft_consensus.cc:3049] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:36.816002 11963 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:36.816761 11963 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:36.817168 11884 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:36.818010 11963 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:36.843446 13516 raft_consensus.cc:988] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:36.844049 13641 raft_consensus.cc:1076] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:36.845963 12042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33876
I20250113 08:58:36.846567 12042 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:36.846946 12042 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:36.852008 11963 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:36.853185 12007 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:36.854317 13640 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:36.854758 13640 raft_consensus.cc:695] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:36.855512 13640 consensus_queue.cc:237] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:36.856280 12042 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:36.860213 11884 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:36.860313 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:36.860966 12726 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:36.861362 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:36.861945 11887 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:58:36.862202 11887 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:36.863123 12042 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:36.863039 11887 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:36.864315 11887 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:36.868880 12726 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:58:36.871028 12008 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:36.872155 13690 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:58:36.872633 13690 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:36.873463 13690 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:36.875450 11887 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:58:36.888537 11788 catalog_manager.cc:5526] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 3 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:36.908844 11788 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 4 to 5, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 5 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:37.089119 13712 ts_tablet_manager.cc:927] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:37.091312 13712 tablet_copy_client.cc:323] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:37.093068 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 043006a99fa54df3a6ee86ebd18003a7 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:37.093677 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 043006a99fa54df3a6ee86ebd18003a7 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-043006a99fa54df3a6ee86ebd18003a7
I20250113 08:58:37.099808 11903 tablet_copy_source_session.cc:215] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:37.102618 13712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043006a99fa54df3a6ee86ebd18003a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:37.104496 13714 ts_tablet_manager.cc:927] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:37.111917 13714 tablet_copy_client.cc:323] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:37.112835 13716 ts_tablet_manager.cc:927] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:37.113411 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 39676b99e4434f11b6017bb867fb7471 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:37.113942 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 39676b99e4434f11b6017bb867fb7471 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-39676b99e4434f11b6017bb867fb7471
I20250113 08:58:37.114554 13716 tablet_copy_client.cc:323] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:37.117632 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 6a1ace8d6d3d488583610962dabdee94 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:37.119133 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 6a1ace8d6d3d488583610962dabdee94 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-6a1ace8d6d3d488583610962dabdee94
I20250113 08:58:37.139091 11903 tablet_copy_source_session.cc:215] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:37.139869 12052 tablet_copy_source_session.cc:215] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:37.141270 13712 tablet_copy_client.cc:806] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:37.141897 13712 tablet_copy_client.cc:670] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:37.142490 13714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39676b99e4434f11b6017bb867fb7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:37.147292 13716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1ace8d6d3d488583610962dabdee94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:37.150477 13712 tablet_copy_client.cc:538] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:37.159572 13714 tablet_copy_client.cc:806] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:37.160187 13714 tablet_copy_client.cc:670] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:37.162420 13712 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:37.164861 13714 tablet_copy_client.cc:538] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:37.177026 13714 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:37.178815 13716 tablet_copy_client.cc:806] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:37.179425 13716 tablet_copy_client.cc:670] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:37.183102 13716 tablet_copy_client.cc:538] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:37.189220 13712 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:37.190177 13712 tablet_bootstrap.cc:492] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:37.190848 13712 ts_tablet_manager.cc:1397] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.007s
I20250113 08:58:37.192790 13716 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:37.195912 13712 raft_consensus.cc:357] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.198141 13712 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:37.198792 13712 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.208720 13714 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:37.209681 13714 tablet_bootstrap.cc:492] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:37.210403 13714 ts_tablet_manager.cc:1397] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.013s
I20250113 08:58:37.213500 13712 ts_tablet_manager.cc:1428] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250113 08:58:37.213061 13714 raft_consensus.cc:357] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.213999 13714 raft_consensus.cc:738] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:37.214659 13714 consensus_queue.cc:260] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.217615 13714 ts_tablet_manager.cc:1428] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250113 08:58:37.219182 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-043006a99fa54df3a6ee86ebd18003a7 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:37.219655 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-043006a99fa54df3a6ee86ebd18003a7 on tablet 043006a99fa54df3a6ee86ebd18003a7 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:37.223865 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-39676b99e4434f11b6017bb867fb7471 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:37.224270 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-39676b99e4434f11b6017bb867fb7471 on tablet 39676b99e4434f11b6017bb867fb7471 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:37.243170 13716 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:37.244138 13716 tablet_bootstrap.cc:492] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:37.244774 13716 ts_tablet_manager.cc:1397] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.052s user 0.020s sys 0.005s
I20250113 08:58:37.247334 13716 raft_consensus.cc:357] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.248178 13716 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:37.248809 13716 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.251091 13716 ts_tablet_manager.cc:1428] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250113 08:58:37.252919 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-6a1ace8d6d3d488583610962dabdee94 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:37.253340 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-6a1ace8d6d3d488583610962dabdee94 on tablet 6a1ace8d6d3d488583610962dabdee94 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:37.256745 12042 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:37.257954 13709 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:37.270790 11887 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:37.270792 12725 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:58:37.273121 13690 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:58:37.278931 13690 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:37.293501 11887 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:58:37.296866 11961 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:37.298377 11887 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:37.300067 13708 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:37.298364 13642 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250113 08:58:37.312917 13642 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:58:37.349845 13716 ts_tablet_manager.cc:927] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:37.351683 13716 tablet_copy_client.cc:323] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:37.354864 11977 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet b6a40dc5e61345558c943df4be928468 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:37.355386 11977 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet b6a40dc5e61345558c943df4be928468 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-b6a40dc5e61345558c943df4be928468
I20250113 08:58:37.364137 11977 tablet_copy_source_session.cc:215] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:37.375598 13716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6a40dc5e61345558c943df4be928468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:37.396890 13716 tablet_copy_client.cc:806] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:37.407573 13716 tablet_copy_client.cc:670] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:37.412900 13716 tablet_copy_client.cc:538] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:37.430359 13716 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:37.465898 13716 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:37.466606 13716 tablet_bootstrap.cc:492] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:37.467183 13716 ts_tablet_manager.cc:1397] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.008s
I20250113 08:58:37.469338 13716 raft_consensus.cc:357] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.470024 13716 raft_consensus.cc:738] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:37.470480 13716 consensus_queue.cc:260] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:37.472718 13716 ts_tablet_manager.cc:1428] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250113 08:58:37.475174 11977 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-b6a40dc5e61345558c943df4be928468 received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:37.475584 11977 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-b6a40dc5e61345558c943df4be928468 on tablet b6a40dc5e61345558c943df4be928468 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:37.547950 12725 raft_consensus.cc:1212] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:37.652760 12725 raft_consensus.cc:1212] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250113 08:58:37.687505 12725 raft_consensus.cc:1212] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250113 08:58:37.737715 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:37.789604 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "364888f946914390a5a07be6735a4cc9"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:37.790127 12037 raft_consensus.cc:604] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:37.793824 11961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:37.794433 11961 raft_consensus.cc:604] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:37.798892 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:37.799386 12037 raft_consensus.cc:604] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:37.800230 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250113 08:58:37.801995 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:37.802632 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:37.940241 13727 raft_consensus.cc:988] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:37.942019 13640 raft_consensus.cc:1076] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:37.944502 11891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52586
I20250113 08:58:37.945046 11891 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:37.945485 11891 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:58:37.949275 12725 raft_consensus.cc:1212] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:37.953995 11891 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:37.957919 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:37.958384 12037 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:58:37.958624 12037 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:37.959221 12037 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:37.960057 12037 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:58:37.960662 11891 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:37.961620 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:37.962172 12725 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:58:37.966149 12037 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:58:37.967319 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:37.967960 12725 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:58:37.968032 13660 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:58:37.968714 13660 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:37.969525 13660 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:37.979995 11787 catalog_manager.cc:5526] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 5 to 6, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 6 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:38.017966 13690 raft_consensus.cc:988] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:38.018437 13640 raft_consensus.cc:1076] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:38.019789 11961 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "364888f946914390a5a07be6735a4cc9"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43872
I20250113 08:58:38.020311 11961 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:38.020720 11961 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:38.026369 11961 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.028589 11961 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:38.028849 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:38.029459 12037 raft_consensus.cc:3049] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:38.029381 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:38.029961 12037 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:38.030148 12725 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:38.030686 12037 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.031661 12037 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:38.037156 12725 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:38.037767 12037 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:38.038163 11934 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:38.039027 13709 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:38.039413 13709 raft_consensus.cc:695] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:38.040166 13709 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.055720 11787 catalog_manager.cc:5526] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:38.069329 13641 raft_consensus.cc:1059] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:38.071769 13641 consensus_queue.cc:237] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.084865 12725 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:38.085542 12037 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:38.087158 13562 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:38.089581 13562 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:38.089090 11962 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:38.093437 13660 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:38.110780 13641 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.118738 11962 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.121043 12037 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.120770 12726 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.131865 11787 catalog_manager.cc:5526] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:38.190585 13640 raft_consensus.cc:1059] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:38.194253 13640 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.203094 11890 consensus_queue.cc:237] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.208237 13594 raft_consensus.cc:988] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:38.208833 13594 raft_consensus.cc:1076] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:38.217855 11891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:38.219087 11891 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:38.224608 11891 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:38.222941 11890 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:38.223615 11962 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:38.225528 12035 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:38.227140 13640 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:38.227730 13746 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:38.228152 13690 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:38.221725 12726 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:38.237710 13640 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:38.238672 13660 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.240404 12035 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.241328 11891 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:38.243078 11891 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:38.246001 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:38.246606 11961 raft_consensus.cc:3049] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:38.246886 11961 raft_consensus.cc:738] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:38.247548 11961 consensus_queue.cc:260] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:38.248654 11961 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:38.249918 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:38.250440 12037 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:38.253635 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 39676b99e4434f11b6017bb867fb7471 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:38.258190 11961 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:38.259325 11858 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:38.259874 12037 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:38.260113 13641 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:38.260659 13641 raft_consensus.cc:695] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:38.260942 12725 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:38.261519 13641 consensus_queue.cc:237] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:38.273525 11788 catalog_manager.cc:5526] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 1 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:38.277702 13562 raft_consensus.cc:1059] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:38.288867 13562 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.299221 13746 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:38.303431 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 39676b99e4434f11b6017bb867fb7471 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:38.307237 12722 raft_consensus.cc:2949] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.313581 13690 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.323343 11884 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.314997 12725 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.344794 12035 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:38.350598 13562 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:38.353190 12725 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:38.354744 13641 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:38.357486 11788 catalog_manager.cc:5526] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.357877 13756 tablet_replica.cc:331] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:38.359772 11961 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:38.362766 13756 raft_consensus.cc:2238] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:38.374400 13756 raft_consensus.cc:2267] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:38.363605 13562 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:38.369833 11790 catalog_manager.cc:5526] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:38.371810 11961 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.378481 13756 ts_tablet_manager.cc:1905] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:38.379915 13754 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.382613 12035 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.395386 12042 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.407415 11962 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.409067 13756 ts_tablet_manager.cc:1918] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:38.409689 13756 log.cc:1198] T 39676b99e4434f11b6017bb867fb7471 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/39676b99e4434f11b6017bb867fb7471
I20250113 08:58:38.411651 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 39676b99e4434f11b6017bb867fb7471 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:38.418390 12722 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:38.418857 12037 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250113 08:58:38.420986 13727 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:38.423333 13745 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250113 08:58:38.427356 12725 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.428748 11887 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:38.430167 13727 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:38.432910 11788 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.438035 13690 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.443637 11884 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.467211 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 6a1ace8d6d3d488583610962dabdee94 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:38.484525 13707 raft_consensus.cc:1059] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:38.488268 13707 consensus_queue.cc:237] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.474076 12725 raft_consensus.cc:2949] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.496500 12722 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250113 08:58:38.491695 11788 catalog_manager.cc:5526] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 4 to 5, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 2 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.501307 13562 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250113 08:58:38.499223 12726 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:38.504117 13594 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:38.555546 12725 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:38.559396 12037 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:38.559279 13709 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:38.560624 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1ace8d6d3d488583610962dabdee94 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:38.561645 13756 tablet_replica.cc:331] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:38.562809 13756 raft_consensus.cc:2238] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:38.563580 13756 raft_consensus.cc:2267] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:38.563768 11884 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:38.566044 13594 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:38.578224 13756 ts_tablet_manager.cc:1905] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:38.589407 13707 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250113 08:58:38.614497 13756 ts_tablet_manager.cc:1918] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250113 08:58:38.614917 13756 log.cc:1198] T 6a1ace8d6d3d488583610962dabdee94 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/6a1ace8d6d3d488583610962dabdee94
I20250113 08:58:38.617895 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 6a1ace8d6d3d488583610962dabdee94 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:38.619164 12037 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:38.618126 13709 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.622545 13707 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:38.623633 11884 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.626695 12722 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.640173 12042 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.645462 11787 catalog_manager.cc:5526] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:38.661044 11961 consensus_queue.cc:237] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.665912 12722 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:38.667748 13709 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250113 08:58:38.671154 12033 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:38.672554 12042 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:38.672556 13746 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:38.675613 13772 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:38.684425 13773 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.697806 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b6a40dc5e61345558c943df4be928468 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:38.698697 12037 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.706879 11961 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:38.712495 13746 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:38.709033 12722 raft_consensus.cc:2949] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.747994 11788 catalog_manager.cc:5526] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:38.763698 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a40dc5e61345558c943df4be928468 with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:38.776827 13777 tablet_replica.cc:331] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:38.777979 13777 raft_consensus.cc:2238] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:38.778687 13777 raft_consensus.cc:2267] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:38.783025 13777 ts_tablet_manager.cc:1905] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:38.802443 13777 ts_tablet_manager.cc:1918] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:38.802806 13777 log.cc:1198] T b6a40dc5e61345558c943df4be928468 P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/b6a40dc5e61345558c943df4be928468
I20250113 08:58:38.804033 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet b6a40dc5e61345558c943df4be928468 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:38.848664 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:38.895363 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:38.896044 11891 raft_consensus.cc:604] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:38.898445 11963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:38.898900 11963 raft_consensus.cc:604] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:38.901423 11891 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:38.902128 11891 raft_consensus.cc:604] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:38.904207 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250113 08:58:38.905992 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:38.906559 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:38.977121 11884 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:38.981509 11963 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:38.981724 12722 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:38.983192 13562 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:38.986228 13746 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:38.997707 13745 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:39.000304 11962 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:39.014043 12722 raft_consensus.cc:2949] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:39.020604 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 043006a99fa54df3a6ee86ebd18003a7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:39.022223 13772 raft_consensus.cc:988] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:39.022855 13709 raft_consensus.cc:1076] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:58:39.031723 12033 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:33878
I20250113 08:58:39.032332 12033 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:39.032653 12033 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:39.029978 11790 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 2 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:39.041839 12033 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:39.044561 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:39.045310 11887 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:39.046175 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:39.046789 11963 raft_consensus.cc:3049] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:39.047077 11963 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:39.048038 12033 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:39.047752 11963 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:39.049420 11963 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:39.052670 11887 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:39.053742 12006 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:39.054761 13766 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:39.054679 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 043006a99fa54df3a6ee86ebd18003a7 with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:39.055311 13766 raft_consensus.cc:695] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:39.056339 13766 consensus_queue.cc:237] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:39.057166 13778 tablet_replica.cc:331] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:39.058439 13778 raft_consensus.cc:2238] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:39.059700 13778 raft_consensus.cc:2267] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:39.061069 11963 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:39.064386 13778 ts_tablet_manager.cc:1905] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:39.073113 11788 catalog_manager.cc:5526] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 4, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:39.079958 13778 ts_tablet_manager.cc:1918] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250113 08:58:39.080461 13778 log.cc:1198] T 043006a99fa54df3a6ee86ebd18003a7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/043006a99fa54df3a6ee86ebd18003a7
I20250113 08:58:39.082352 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet 043006a99fa54df3a6ee86ebd18003a7 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:39.111042 13745 raft_consensus.cc:988] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:39.111585 13745 raft_consensus.cc:1076] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:39.112931 12722 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38504
I20250113 08:58:39.113523 12722 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:39.113864 12722 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:39.119779 12722 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:39.121776 12722 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:39.123457 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:39.124043 11887 raft_consensus.cc:3049] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:39.124341 11887 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:39.124995 11887 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:39.125517 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:39.126356 11887 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:39.126518 11962 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:39.132541 11887 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:39.132645 11962 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:39.133438 12685 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:39.134079 13662 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:39.134435 13662 raft_consensus.cc:695] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:39.135181 13662 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:39.142868 11788 catalog_manager.cc:5526] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 3, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 3 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:39.155616 13754 raft_consensus.cc:988] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:39.156131 13745 raft_consensus.cc:1076] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:39.157737 11962 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:39.158566 11962 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:39.158860 11962 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:39.163376 11962 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:39.165205 11962 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:39.165784 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:39.166297 12035 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:39.165629 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:39.166803 11887 raft_consensus.cc:3049] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:39.167100 11887 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:39.167785 11887 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:39.168964 11887 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:39.172787 12035 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:39.173951 11931 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:39.174669 13709 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:39.175045 13709 raft_consensus.cc:695] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:39.175340 11887 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:39.175838 13709 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:39.186285 11788 catalog_manager.cc:5526] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:39.545086 11962 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:58:39.546244 13766 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:39.562322 11891 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:39.564440 13709 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:39.565075 11887 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:58:39.585976 13640 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:39.627092 11884 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:39.652184 12035 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:39.635546 13581 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20250113 08:58:39.666646 13794 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:39.702473 11963 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:39.704324 13793 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:39.958698 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:39.994537 12035 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
mode: GRACEFUL
new_leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:42362
I20250113 08:58:39.995096 12035 raft_consensus.cc:604] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Received request to transfer leadership to TS 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:39.997408 11962 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:39.997877 11962 raft_consensus.cc:604] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:39.999043 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250113 08:58:40.000387 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:40.000989 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:40.043081 13766 raft_consensus.cc:988] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: : Instructing follower 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:40.043545 13783 raft_consensus.cc:1076] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Signalling peer 5f9c33777f6845c2848a2a6102a1c2ff to start an election
I20250113 08:58:40.053885 12726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
from {username='slave'} at 127.0.0.1:38520
I20250113 08:58:40.054451 12726 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:40.054809 12726 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:40.061915 12726 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:40.063453 13794 raft_consensus.cc:988] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:40.063949 13796 raft_consensus.cc:1076] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:40.064728 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:40.064879 12726 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:40.065361 11884 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:40.066063 11891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:40.066614 11891 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:40.066931 11891 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:40.070518 11884 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:40.070399 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:40.071085 12035 raft_consensus.cc:3049] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:40.071370 12035 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:40.071588 12685 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:40.072046 12035 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:40.072850 13581 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:40.074071 12035 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:40.076020 13581 raft_consensus.cc:695] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:40.076150 11891 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:40.076882 13581 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:40.080422 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:40.079038 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:40.081535 12726 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:40.079528 11891 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:40.081104 11962 raft_consensus.cc:3049] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:40.086611 11962 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:40.087425 11962 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:40.088483 11962 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:40.096778 11962 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:40.097752 12726 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:40.098995 11859 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:40.099788 13562 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:40.100240 13562 raft_consensus.cc:695] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:40.101037 13562 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:40.106392 12035 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:40.122457 11787 catalog_manager.cc:5526] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 3, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 3 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:40.123354 11790 catalog_manager.cc:5526] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:40.130473 11890 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.139974 12033 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250113 08:58:40.139994 11962 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250113 08:58:40.141948 13745 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:40.143813 13745 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250113 08:58:40.151520 11859 consensus_peers.cc:487] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7104dce7b01a423493a37b6ed694da07. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:40.153192 13562 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.158622 11961 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.180590 12035 consensus_queue.cc:237] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.183619 12033 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.183540 11790 catalog_manager.cc:5526] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 4, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
W20250113 08:58:40.188297 12008 consensus_peers.cc:487] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 663c56b937924e7abebd7025ca892c1e. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:40.193397 11884 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:40.194996 13783 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:40.196398 11961 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:58:40.201339 13783 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:40.243162 13766 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.252539 11962 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.256408 11887 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.273387 11787 catalog_manager.cc:5526] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.281315 11961 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:40.320602 11934 consensus_peers.cc:487] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a43de3c8f06b48358bcee58f9567bb3b. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:40.322603 12035 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:40.323952 11884 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:40.325232 13812 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:40.326730 13796 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:40.336467 13796 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.338577 11884 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.339061 12035 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.354678 12033 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.355760 11787 catalog_manager.cc:5526] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.364043 11961 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250113 08:58:40.366207 13766 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250113 08:58:40.367188 12008 consensus_peers.cc:487] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce0640fd1a5740f1ac76ebcae33dfa25. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:40.373346 11884 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250113 08:58:40.374441 13766 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:40.382683 11963 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.380743 13814 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.408123 11790 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 5, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.412604 11884 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:40.525314 11884 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:40.526441 13793 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:40.530427 12726 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:40.533005 13562 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:40.540712 11963 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:40.542411 13745 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:40.563185 12035 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:40.564519 13793 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:40.628962 13830 ts_tablet_manager.cc:927] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:40.656541 13830 tablet_copy_client.cc:323] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:40.658617 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet 663c56b937924e7abebd7025ca892c1e from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:40.659193 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet 663c56b937924e7abebd7025ca892c1e from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-663c56b937924e7abebd7025ca892c1e
I20250113 08:58:40.667091 12052 tablet_copy_source_session.cc:215] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:40.675944 13831 ts_tablet_manager.cc:927] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539)
I20250113 08:58:40.679368 13831 tablet_copy_client.cc:323] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.129:33539
I20250113 08:58:40.680914 11903 tablet_copy_service.cc:140] P 6ea00811b7d44b49b872ac5c03b91487: Received BeginTabletCopySession request for tablet 7104dce7b01a423493a37b6ed694da07 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:57686)
I20250113 08:58:40.681402 11903 tablet_copy_service.cc:161] P 6ea00811b7d44b49b872ac5c03b91487: Beginning new tablet copy session on tablet 7104dce7b01a423493a37b6ed694da07 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:57686: session id = 5f9c33777f6845c2848a2a6102a1c2ff-7104dce7b01a423493a37b6ed694da07
I20250113 08:58:40.687841 13830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663c56b937924e7abebd7025ca892c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:40.692688 11903 tablet_copy_source_session.cc:215] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:40.696465 13831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7104dce7b01a423493a37b6ed694da07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:40.719633 13831 tablet_copy_client.cc:806] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:40.720394 13831 tablet_copy_client.cc:670] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:40.723999 13831 tablet_copy_client.cc:538] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:40.734309 13831 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:40.742872 13830 tablet_copy_client.cc:806] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:40.743589 13830 tablet_copy_client.cc:670] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:40.748775 13830 tablet_copy_client.cc:538] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:40.760846 13830 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:40.767340 13831 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:40.768299 13831 tablet_bootstrap.cc:492] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:40.769094 13831 ts_tablet_manager.cc:1397] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.004s
I20250113 08:58:40.771991 13831 raft_consensus.cc:357] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.773001 13831 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:40.773744 13831 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.776543 13831 ts_tablet_manager.cc:1428] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250113 08:58:40.781275 11903 tablet_copy_service.cc:342] P 6ea00811b7d44b49b872ac5c03b91487: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-7104dce7b01a423493a37b6ed694da07 received from {username='slave'} at 127.0.0.1:57686
I20250113 08:58:40.781684 11903 tablet_copy_service.cc:434] P 6ea00811b7d44b49b872ac5c03b91487: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-7104dce7b01a423493a37b6ed694da07 on tablet 7104dce7b01a423493a37b6ed694da07 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:40.798861 13830 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:40.799834 13830 tablet_bootstrap.cc:492] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:40.800592 13830 ts_tablet_manager.cc:1397] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.040s user 0.021s sys 0.011s
I20250113 08:58:40.803501 13830 raft_consensus.cc:357] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.804486 13830 raft_consensus.cc:738] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:40.805184 13830 consensus_queue.cc:260] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.809556 13830 ts_tablet_manager.cc:1428] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250113 08:58:40.811480 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-663c56b937924e7abebd7025ca892c1e received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:40.811932 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-663c56b937924e7abebd7025ca892c1e on tablet 663c56b937924e7abebd7025ca892c1e with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:40.848065 13830 ts_tablet_manager.cc:927] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:40.849731 13830 tablet_copy_client.cc:323] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:40.851081 11977 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet a43de3c8f06b48358bcee58f9567bb3b from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:40.851652 11977 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet a43de3c8f06b48358bcee58f9567bb3b from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-a43de3c8f06b48358bcee58f9567bb3b
I20250113 08:58:40.860163 11977 tablet_copy_source_session.cc:215] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:40.867274 13830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43de3c8f06b48358bcee58f9567bb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:40.883337 13830 tablet_copy_client.cc:806] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:40.883934 13830 tablet_copy_client.cc:670] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:40.889278 13830 tablet_copy_client.cc:538] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:40.898422 13830 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:40.921769 13830 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:40.922766 13830 tablet_bootstrap.cc:492] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:40.923410 13830 ts_tablet_manager.cc:1397] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.003s
I20250113 08:58:40.926376 13830 raft_consensus.cc:357] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.927294 13830 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:40.927820 13830 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:40.930222 13830 ts_tablet_manager.cc:1428] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250113 08:58:40.932124 11977 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a43de3c8f06b48358bcee58f9567bb3b received from {username='slave'} at 127.0.0.1:54924
I20250113 08:58:40.932543 11977 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-a43de3c8f06b48358bcee58f9567bb3b on tablet a43de3c8f06b48358bcee58f9567bb3b with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:40.950927 13830 ts_tablet_manager.cc:927] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:40.952780 13830 tablet_copy_client.cc:323] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.131:43231
I20250113 08:58:40.954994 12052 tablet_copy_service.cc:140] P 5e8883f5f83a47f4b9303dd1da4f59c0: Received BeginTabletCopySession request for tablet ce0640fd1a5740f1ac76ebcae33dfa25 from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:37184)
I20250113 08:58:40.955567 12052 tablet_copy_service.cc:161] P 5e8883f5f83a47f4b9303dd1da4f59c0: Beginning new tablet copy session on tablet ce0640fd1a5740f1ac76ebcae33dfa25 from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:37184: session id = 5f9c33777f6845c2848a2a6102a1c2ff-ce0640fd1a5740f1ac76ebcae33dfa25
I20250113 08:58:40.964617 12052 tablet_copy_source_session.cc:215] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:40.968111 13830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0640fd1a5740f1ac76ebcae33dfa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:40.989545 13830 tablet_copy_client.cc:806] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:40.990146 13830 tablet_copy_client.cc:670] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:40.993835 13830 tablet_copy_client.cc:538] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:41.004101 13830 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:41.026721 12726 raft_consensus.cc:1212] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20250113 08:58:41.042377 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:41.055501 13830 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:58:41.056489 13830 tablet_bootstrap.cc:492] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:58:41.057204 13830 ts_tablet_manager.cc:1397] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 0.053s user 0.025s sys 0.028s
I20250113 08:58:41.060109 13830 raft_consensus.cc:357] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:41.060962 13830 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:58:41.061604 13830 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:41.064240 13830 ts_tablet_manager.cc:1428] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250113 08:58:41.074427 12052 tablet_copy_service.cc:342] P 5e8883f5f83a47f4b9303dd1da4f59c0: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-ce0640fd1a5740f1ac76ebcae33dfa25 received from {username='slave'} at 127.0.0.1:37184
I20250113 08:58:41.074895 12052 tablet_copy_service.cc:434] P 5e8883f5f83a47f4b9303dd1da4f59c0: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-ce0640fd1a5740f1ac76ebcae33dfa25 on tablet ce0640fd1a5740f1ac76ebcae33dfa25 with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:41.104882 11887 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:58:41.105551 11887 raft_consensus.cc:604] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:41.106611 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250113 08:58:41.108390 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:41.109002 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:41.146903 13826 raft_consensus.cc:988] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: : Instructing follower 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:41.147377 13754 raft_consensus.cc:1076] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Signalling peer 37bd4bed3ee6423c8dac0cef3bf6882a to start an election
I20250113 08:58:41.148921 11963 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43888
I20250113 08:58:41.149408 11963 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:41.149674 11963 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:41.160794 11963 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.162803 11963 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:41.163259 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:41.163879 11884 raft_consensus.cc:3049] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:41.164084 11884 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:41.163643 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:41.164727 12726 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:41.164580 11884 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.165901 11884 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:41.170284 12726 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:41.173034 11934 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:41.173794 13796 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:41.174221 13796 raft_consensus.cc:695] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:41.174925 13796 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.178339 11884 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:41.189201 12725 raft_consensus.cc:1212] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250113 08:58:41.187383 11787 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 2 to 3, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:41.357662 12726 raft_consensus.cc:1212] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
I20250113 08:58:41.437393 12726 raft_consensus.cc:1212] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250113 08:58:41.476812 13754 raft_consensus.cc:1059] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:41.479384 13754 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.484854 12726 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEARNER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:41.485102 12037 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:41.486179 11963 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:41.487287 13826 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:41.488298 13562 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:41.491925 13754 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:41.516664 13754 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.518335 11961 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.527489 12726 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.544206 12037 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.547984 11787 catalog_manager.cc:5526] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 4 to 5, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:41.563148 11887 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.568773 12726 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:41.569581 12033 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:41.570122 13562 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:41.571671 13754 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:41.577406 13826 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.581974 12726 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.584031 12037 raft_consensus.cc:2949] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.586822 11776 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7104dce7b01a423493a37b6ed694da07 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:41.607349 11788 catalog_manager.cc:5526] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 5 to 6, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:41.620126 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet 7104dce7b01a423493a37b6ed694da07 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:41.623028 13850 tablet_replica.cc:331] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:41.624135 13850 raft_consensus.cc:2238] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:41.625039 13850 raft_consensus.cc:2267] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:41.628664 13850 ts_tablet_manager.cc:1905] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:41.644309 13850 ts_tablet_manager.cc:1918] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20250113 08:58:41.644693 13850 log.cc:1198] T 7104dce7b01a423493a37b6ed694da07 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/7104dce7b01a423493a37b6ed694da07
I20250113 08:58:41.646368 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet 7104dce7b01a423493a37b6ed694da07 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:41.660964 13813 raft_consensus.cc:1059] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:41.663017 13813 consensus_queue.cc:237] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.671880 11887 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:41.673630 13845 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:41.676208 12726 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:41.677845 13766 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:41.687333 11961 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:41.689168 11887 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 08:58:41.690310 13766 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:58:41.691256 13841 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:58:41.710944 12725 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:41.714697 13796 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250113 08:58:41.758749 13859 log.cc:926] Time spent T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250113 08:58:41.761385 13857 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.764197 11890 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.804595 12726 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.807616 11788 catalog_manager.cc:5526] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.813390 11963 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.844550 12033 consensus_queue.cc:237] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.849220 12725 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:41.849355 11963 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250113 08:58:41.850956 13766 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:41.853374 13766 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:41.866614 13862 raft_consensus.cc:1059] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:41.869191 13862 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:41.869817 13841 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.876474 11963 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.881042 12725 raft_consensus.cc:2949] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.893813 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 663c56b937924e7abebd7025ca892c1e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:41.895720 11789 catalog_manager.cc:5526] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 3 to 4, VOTER 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) evicted. New cstate: current_term: 1 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.904389 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 663c56b937924e7abebd7025ca892c1e with delete_type TABLET_DATA_TOMBSTONED (TS 6ea00811b7d44b49b872ac5c03b91487 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20250113 08:58:41.928223 13864 tablet_replica.cc:331] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: stopping tablet replica
I20250113 08:58:41.932828 13864 raft_consensus.cc:2238] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:41.938886 13864 raft_consensus.cc:2267] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:41.940999 11887 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:41.941694 11962 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:41.942507 13862 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:41.945209 12725 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:41.946632 13841 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:41.947629 13862 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:41.949000 13864 ts_tablet_manager.cc:1905] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:41.969508 13841 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.971246 11962 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.972752 13864 ts_tablet_manager.cc:1918] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250113 08:58:41.973132 13864 log.cc:1198] T 663c56b937924e7abebd7025ca892c1e P 6ea00811b7d44b49b872ac5c03b91487: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-0-root/wals/663c56b937924e7abebd7025ca892c1e
I20250113 08:58:41.975222 11776 catalog_manager.cc:4872] TS 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): tablet 663c56b937924e7abebd7025ca892c1e (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:41.974337 11890 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.987470 12725 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:41.998111 11789 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index 5 to 6, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } attrs { replace: true } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.015731 12035 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:42.021629 12726 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250113 08:58:42.023021 13841 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:58:42.025117 11887 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250113 08:58:42.026516 13841 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:58:42.028738 13794 raft_consensus.cc:1059] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:58:42.032115 13794 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:42.034840 13766 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.038444 12726 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.046645 11775 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ce0640fd1a5740f1ac76ebcae33dfa25 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:42.047847 12725 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEARNER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:42.049039 11890 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:42.049875 12035 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250113 08:58:42.050726 13796 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:42.052677 13796 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:42.060505 13845 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:42.060923 11888 raft_consensus.cc:2949] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.061875 11788 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: config changed from index 6 to 7, VOTER 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) evicted. New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:42.076035 11950 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0640fd1a5740f1ac76ebcae33dfa25 with delete_type TABLET_DATA_TOMBSTONED (TS 37bd4bed3ee6423c8dac0cef3bf6882a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:43866
I20250113 08:58:42.077111 13850 tablet_replica.cc:331] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: stopping tablet replica
I20250113 08:58:42.108470 13845 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.110675 11890 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.113755 13850 raft_consensus.cc:2238] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:42.114933 13850 raft_consensus.cc:2267] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:42.119318 13850 ts_tablet_manager.cc:1905] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:42.124437 12725 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.133783 11787 catalog_manager.cc:5526] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.146219 12033 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.160873 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:42.184134 13850 ts_tablet_manager.cc:1918] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20250113 08:58:42.184744 13850 log.cc:1198] T ce0640fd1a5740f1ac76ebcae33dfa25 P 37bd4bed3ee6423c8dac0cef3bf6882a: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-1-root/wals/ce0640fd1a5740f1ac76ebcae33dfa25
I20250113 08:58:42.185086 11963 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:42.190321 11777 catalog_manager.cc:4872] TS 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): tablet ce0640fd1a5740f1ac76ebcae33dfa25 (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:42.192015 12725 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:42.194043 13869 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:42.195616 11888 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:42.196867 13794 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:42.201534 13845 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.207008 12725 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.220736 11888 raft_consensus.cc:2949] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.254438 11777 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a43de3c8f06b48358bcee58f9567bb3b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250113 08:58:42.258020 11790 catalog_manager.cc:5526] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: config changed from index 4 to 5, VOTER 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) evicted. New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:42.274871 11962 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54973520c9884989a12742d4eb45994f"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:42.275496 11962 raft_consensus.cc:604] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:42.277489 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250113 08:58:42.281152 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet a43de3c8f06b48358bcee58f9567bb3b with delete_type TABLET_DATA_TOMBSTONED (TS 5e8883f5f83a47f4b9303dd1da4f59c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33860
I20250113 08:58:42.285431 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:42.286842 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:42.290167 13877 tablet_replica.cc:331] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: stopping tablet replica
I20250113 08:58:42.290939 13877 raft_consensus.cc:2238] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 08:58:42.291548 13877 raft_consensus.cc:2267] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 08:58:42.294152 13877 ts_tablet_manager.cc:1905] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 08:58:42.306622 13877 ts_tablet_manager.cc:1918] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250113 08:58:42.307024 13877 log.cc:1198] T a43de3c8f06b48358bcee58f9567bb3b P 5e8883f5f83a47f4b9303dd1da4f59c0: Deleting WAL directory at /tmp/dist-test-taskQpDTIx/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736758601939195-10702-0/minicluster-data/ts-2-root/wals/a43de3c8f06b48358bcee58f9567bb3b
I20250113 08:58:42.308413 11775 catalog_manager.cc:4872] TS 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): tablet a43de3c8f06b48358bcee58f9567bb3b (table test-workload [id=2af8d02864c54b3dbd3bdd894f76ddf5]) successfully deleted
I20250113 08:58:43.382055 13880 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.382719 13880 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.385679 13880 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.420873 13887 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.423430 13887 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.431196 13887 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.470964 13878 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.471537 13878 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.477952 13878 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.498427 13880 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.499944 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:43.606094 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 120620us
I20250113 08:58:43.557664 13880 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.653043 13899 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.653223 13904 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.653614 13899 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.654181 13880 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.654109 13904 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.644068 13881 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.655050 13888 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.655170 13881 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:43.649047 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 90299.9us
I20250113 08:58:43.649400 13896 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.651974 13907 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.656363 13891 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.656778 13914 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.656929 13907 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.644564 13886 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.657383 13914 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.657223 13891 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.658124 13886 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.659092 13907 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.656239 13904 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.512885 13905 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.660082 13905 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.662103 13905 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.653517 13903 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.662678 13896 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.664678 13882 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.665510 13896 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.665225 13882 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.663321 13903 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.674469 13906 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.674705 13914 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.655764 13899 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.675354 13881 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.675021 13906 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.676437 13882 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.681516 13906 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:43.679726 13901 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.682500 13908 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.687887 13894 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.688477 13894 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.676081 13888 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.682219 13901 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.707837 13888 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.700520 13903 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.689033 13891 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.714633 13894 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.717913 13901 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:43.700558 13886 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.718504 13906 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.719290 13906 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.726006 13919 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.726670 13919 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.727646 13894 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.728375 13894 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.729800 13919 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.731194 13894 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:43.732023 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 63657.7us
I20250113 08:58:43.705655 13913 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.727696 13916 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.737600 13915 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.738062 13915 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.737728 13916 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.690548 13908 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.739125 13913 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.741747 13908 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:43.742008 13916 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.742113 13915 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.742161 13913 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.740474 13906 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.777099 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:43.779369 11891 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
I20250113 08:58:43.786072 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:43.787312 11888 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:58:43.789288 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:43.791119 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:43.807312 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:43.810243 13880 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.810470 13888 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.811432 13890 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.813964 13889 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.822913 13883 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.814816 13891 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.818253 13878 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.830227 13878 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.819236 13887 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.825531 13891 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.833817 13887 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.835745 13891 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.836694 13887 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.837579 13880 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.827139 13886 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.843878 13880 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.844678 13880 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.844486 13886 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.834233 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:43.827999 13879 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.824028 13888 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.839803 13884 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.845737 13880 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.851225 13906 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:43.853642 13903 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.823427 13889 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.875958 13889 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:43.849040 13886 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.847083 13879 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.848855 13878 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.823261 13890 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.863133 13888 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:43.823801 13883 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:43.906100 13892 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.879624 13891 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.879015 13887 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.907116 13892 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.916174 11962 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:43.993362 13909 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:43.888998 13886 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.066712 13918 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.046854 13880 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.082301 13913 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:44.037750 13878 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.046474 13899 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.915180 13906 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:43.982669 13903 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.303021 13935 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.303706 13888 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.082818 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:44.082137 13902 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:43.989367 13894 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.955737 13920 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.082520 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:43.927204 13901 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.207285 13908 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.304579 13888 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.026178 13910 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.019861 13885 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:43.927505 13915 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.260888 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:44.202409 13890 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.063410 13884 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.288857 13926 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.303956 13935 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.116183 13880 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:44.288631 13905 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.248955 13881 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.176136 13912 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.334455 13888 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.083015 13898 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.260241 13896 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.207512 13882 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.289070 13916 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.175823 13883 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.351085 13924 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.217103 13932 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.320612 13879 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.366896 13894 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.287110 13886 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.261006 13917 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.259837 13938 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:44.082199 13923 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.234440 13887 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.240695 13891 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.342010 11961 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:44.221220 13929 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.325973 13925 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.352252 13880 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.358095 13922 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.358691 13936 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.343452 13913 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.334857 13920 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.359504 13934 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.224607 13909 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.360195 13896 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.343753 13899 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.361413 13901 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.362511 13878 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.267259 13918 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.382053 13897 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.382362 13927 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.389822 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:44.361775 13885 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.389495 13898 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.397554 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.393836 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.382089 13941 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.397400 13938 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.357734 13902 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.374666 13908 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.398541 13915 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.424093 13880 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.724431 13927 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.726145 13947 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.727159 13927 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.726864 13947 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.731346 13940 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.511584 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.417098 13916 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.424028 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:44.437688 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08002fbca0 after lost signal to thread 10705
I20250113 08:58:44.766253 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.770510 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:44.437323 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 263595us
I20250113 08:58:44.424160 13899 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.765130 13945 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.808032 13920 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.584882 13946 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.654667 13903 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.603587 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.614495 13892 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.446334 13906 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.817648 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.764958 13943 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.799614 13886 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.441660 13939 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.799065 13894 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.438016 13942 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.414125 13881 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.731819 13941 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.448243 13882 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.836310 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.437101 13931 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:44.426390 13905 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.764971 13950 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:44.820941 13935 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.807528 13945 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.506829 13932 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.774752 13897 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.532742 13924 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.844905 13886 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.845885 12685 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:44.847080 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:44.847157 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:44.408492 13910 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.827471 13928 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.834234 13887 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.850528 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.795532 13933 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:44.853915 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.535562 13934 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.439810 13929 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.861057 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:44.512662 13926 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.773833 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.446707 13917 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.870682 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:44.858021 13891 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:44.869709 12747 debug-util.cc:398] Leaking SignalData structure 0x7b0800280e80 after lost signal to thread 11825
I20250113 08:58:44.441900 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.872943 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.585469 13922 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.634397 13936 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.876821 13884 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.426440 13912 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.864730 13913 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.875252 12725 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:44.732317 13940 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.869016 13947 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.869866 13909 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.890713 11888 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:44.439517 13923 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.877426 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.866595 13886 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.876386 13931 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.899194 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.556293 13925 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.861613 13939 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.886269 13901 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.905684 11891 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:44.856104 13946 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.888875 13903 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.835500 13943 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.916095 13882 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.918116 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.911590 12685 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:44.889461 12720 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:44.898860 13953 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:44.904491 13908 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.902449 13947 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.923729 13898 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.923903 13912 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.914255 13878 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.919090 13918 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.927733 13938 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.932183 13934 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.918946 13916 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.920598 13909 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.921901 13885 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.925428 13915 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.933476 13882 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:44.926753 13945 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.912480 13949 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.938020 13902 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.877315 13950 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.890168 13933 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.950644 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.889804 13928 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:44.956269 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08002af2c0 after lost signal to thread 11917
I20250113 08:58:44.876314 13942 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.966699 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:44.966889 13916 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.967662 13916 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.966975 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.944854 13918 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:44.970669 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.974066 11934 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:44.975245 13953 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.976518 13953 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:44.975831 13885 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:44.973075 13941 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.971778 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.980295 13885 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:44.936218 13903 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.945921 13882 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:44.981976 13881 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:44.981293 12007 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:44.987071 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.985983 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.989670 12725 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:44.971719 13918 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.953186 13909 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:44.990614 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:44.992349 12724 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:44.984546 13885 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:44.995740 11887 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:44.951992 13947 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:44.988139 13882 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:44.999753 13908 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:44.999990 13936 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.000228 13908 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:45.000785 13908 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:45.002800 13916 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.005720 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.005419 13932 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.987857 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:45.007856 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 206468us
W20250113 08:58:45.009902 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08002c26c0 after lost signal to thread 11992
I20250113 08:58:44.993580 13918 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:44.993286 13950 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.996639 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.016177 11934 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:44.993872 13924 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.017153 13923 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:44.997421 13926 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.020622 13929 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.020406 13922 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.030243 13917 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.031219 13925 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.007522 13943 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.008204 13931 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.033188 12007 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:45.034013 13902 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:45.034463 13902 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:45.034874 13902 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:45.035158 13918 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:58:45.038841 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08001bb1a0 after lost signal to thread 12066
I20250113 08:58:45.013834 13910 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.018329 13909 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.025326 13886 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.051309 12008 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:45.053489 13939 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:45.083305 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 46113.3us
I20250113 08:58:45.093834 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.097812 13905 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.098760 13940 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.106523 13926 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:45.107051 13926 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:45.107399 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:45.108274 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 85951.5us
I20250113 08:58:45.107419 13926 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:44.995325 13949 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:45.149665 13912 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:45.150466 13912 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.010000 13897 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.201341 13936 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:45.202178 13936 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.211520 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.213812 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.215620 13936 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.215857 12721 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
W20250113 08:58:45.222352 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 171252us
I20250113 08:58:45.223805 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.224298 13931 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:45.224697 13931 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:45.224732 13947 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.225113 13931 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:45.225901 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.035717 13918 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.231508 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.233693 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.236718 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.227031 13903 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.237866 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.241628 12685 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:45.244266 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.244267 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.244858 12685 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:45.229461 13882 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:45.246737 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 213677us
I20250113 08:58:45.246309 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.247565 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.247959 11887 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:45.252135 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.253906 13936 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:45.254083 13912 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.254432 13936 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:45.254892 13936 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:45.257591 13933 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.259521 11934 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:58:45.269145 12747 debug-util.cc:398] Leaking SignalData structure 0x7b08001a97e0 after lost signal to thread 12750
I20250113 08:58:45.269682 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.269404 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.269483 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.269912 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.275256 11887 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
W20250113 08:58:45.275314 12684 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 1.592s (SENT) [suppressed 4 similar messages]
I20250113 08:58:45.278335 13943 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:45.279129 13943 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.270435 13939 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:45.280938 13939 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:45.281463 13939 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:45.292450 13923 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:45.292564 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.292938 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.293154 13923 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:45.295514 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.295730 13912 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:45.296202 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.296940 11961 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
I20250113 08:58:45.307257 13883 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:45.308033 13883 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:45.309692 13946 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.310809 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.311486 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.312382 12037 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:45.311834 13942 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.258921 13928 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.261278 13949 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.296221 13912 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:45.314750 13912 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:45.301601 13896 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.322682 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.295662 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:45.368402 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 52532.1us
I20250113 08:58:45.375516 12008 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:45.383190 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.390373 12685 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:45.391471 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.362076 13948 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:45.392268 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.393676 13915 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:45.402091 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.393942 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.402527 13915 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:45.402798 11884 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:45.403121 13915 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:45.393748 13943 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.398854 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.401115 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.406195 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.401310 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.401486 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.386114 13946 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:45.397873 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:45.408737 11890 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:45.413086 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.413506 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.413153 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.414589 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.415299 12723 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:45.414523 13931 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.415992 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.416698 12033 raft_consensus.cc:2388] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:45.413486 13882 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.408025 13946 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:45.418186 13946 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:45.419763 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.386430 13918 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.413488 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.424530 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.413089 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.426750 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.432176 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.432353 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.434250 12726 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:58:45.427297 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.437148 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.437906 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.441756 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:45.439752 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 164564us
I20250113 08:58:45.443089 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.448730 12008 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:45.468066 13896 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:45.468554 13896 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:45.468945 13896 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:45.458415 13883 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.475126 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.447878 13931 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.478411 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:45.445117 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.469373 13923 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.480659 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.482580 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.483211 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.485057 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.490496 11890 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:58:45.492496 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.493180 11887 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:58:45.436818 13948 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.494432 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.498178 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.500613 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.501503 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.488693 12720 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:58:45.505188 12033 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:45.497689 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.499392 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.509908 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.509989 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.512246 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.512715 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.513507 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.479451 13917 consensus_queue.cc:1046] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20250113 08:58:45.516054 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.516201 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.516911 12008 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:45.519927 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.520705 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.521461 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.522964 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:45.507555 13953 consensus_meta.cc:287] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.531s user 0.046s sys 0.032s
I20250113 08:58:45.521147 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.529214 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.530328 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.529929 12039 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:45.537539 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.534317 13901 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:58:45.545233 12685 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:45.545933 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.546555 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.550109 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.530776 13953 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.554921 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:58:45.554265 13908 consensus_meta.cc:287] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.552s user 0.011s sys 0.012s
I20250113 08:58:45.552515 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.558107 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.559238 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.560635 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.561340 12042 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:45.562112 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.564286 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.563154 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.574509 13950 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:58:45.575048 13950 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:45.575273 12036 raft_consensus.cc:1235] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:45.575974 13950 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.550050 13918 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:58:45.563604 13908 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.578420 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.579290 13908 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.544852 13901 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:45.579546 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.579979 13901 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:58:45.580127 13902 consensus_meta.cc:287] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.545s user 0.013s sys 0.008s
I20250113 08:58:45.580664 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.563731 12034 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:45.582020 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.583076 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.583385 12035 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:45.576443 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.583639 12039 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:45.580585 13902 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.584895 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.584110 12038 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
I20250113 08:58:45.587674 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.598718 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.588063 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.611986 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.587750 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.586270 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.590219 13915 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.615880 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.617993 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.613479 11968 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
I20250113 08:58:45.621090 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.619988 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.587467 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.628029 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:45.628495 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.631698 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.632514 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.632685 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:45.629998 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 78605.9us
I20250113 08:58:45.633523 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.587975 13950 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:45.599269 13918 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:45.645296 12034 raft_consensus.cc:1235] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
W20250113 08:58:45.646813 13885 consensus_meta.cc:287] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.637s user 0.021s sys 0.013s
I20250113 08:58:45.660676 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.660778 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.646641 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.647603 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.647712 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:45.648428 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.648456 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:45.651093 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.652506 13953 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.630491 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:45.620726 13948 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.655161 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.655417 12717 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:45.655741 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.655879 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.656594 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.658178 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.682410 12035 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
I20250113 08:58:45.655879 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:45.659655 13926 consensus_meta.cc:287] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.533s user 0.013s sys 0.008s
I20250113 08:58:45.689204 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:45.693701 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.695969 13943 consensus_queue.cc:1046] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:45.697114 13943 raft_consensus.cc:3049] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:45.697470 13943 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:45.698423 13943 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.699625 13943 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:45.703233 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.704176 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.704443 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.706058 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:45.592764 13939 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.597116 13936 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.627126 13896 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.674171 12686 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.675940 13946 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.677349 11932 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:45.714241 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 47039.2us
I20250113 08:58:45.666416 13885 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.714784 13953 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:45.722157 12686 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:45.722321 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.723485 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.724284 13897 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:45.724675 13897 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:45.725222 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.725529 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.726444 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.688865 13926 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.727571 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.728432 13926 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.728721 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.725226 13897 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:45.730214 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:45.733503 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 109002us
I20250113 08:58:45.734493 13923 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:45.734865 13923 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:45.735244 13923 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:45.735402 11932 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:45.715335 12007 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:45.738131 12007 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:45.720734 13896 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.729911 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.724598 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:45.742100 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:45.742571 11963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:45.743366 11963 raft_consensus.cc:604] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.744493 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.741443 12686 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.747133 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.745777 11932 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.670598 13915 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.748135 12037 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:45.749369 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.750310 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:45.751626 11962 raft_consensus.cc:1235] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:45.756943 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.753065 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.753120 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.752435 12686 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.745296 13933 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:45.756474 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.760706 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.745653 13926 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:45.765969 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.760092 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250113 08:58:45.766623 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.767082 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.761297 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.758495 12721 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:45.768317 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:45.761883 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.763352 13957 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:45.769496 11967 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:45.770612 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:45.763584 13948 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:45.757090 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:45.773098 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.776157 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.779213 13939 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.779628 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.780601 12006 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.777184 12038 raft_consensus.cc:1235] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:45.782159 13957 consensus_queue.cc:1046] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:45.782454 12006 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.761999 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:45.765720 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:45.749405 13910 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:45.785012 13910 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:45.785465 13910 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:45.782968 13958 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:45.784904 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.783735 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.784579 13898 raft_consensus.cc:3049] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:45.802230 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:45.802953 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.757417 13956 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:45.796005 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:45.802160 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:45.825510 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 89017.7us
I20250113 08:58:45.753352 12007 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:45.798660 12042 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:45.828394 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:45.828635 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 46253.6us
I20250113 08:58:45.830350 12006 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:58:45.832655 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 165895us
I20250113 08:58:45.788472 13939 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:45.799938 13933 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:45.800038 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:45.801482 13936 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:45.836963 12006 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.826540 13902 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.826956 13898 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:58:45.838723 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 85496.8us
I20250113 08:58:45.847265 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.848344 12007 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:45.761818 13912 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.830662 13923 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:45.782984 13901 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.801764 13948 consensus_queue.cc:1046] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20250113 08:58:45.836277 13958 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:45.836602 13939 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:45.838177 13925 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:45.789153 13918 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.881600 13933 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:45.884538 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.886193 12033 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:45.886015 13898 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.875691 13926 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:45.887881 13923 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:45.860945 12684 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:45.864913 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:46.058450 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 224223us
W20250113 08:58:45.923331 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 87961us
I20250113 08:58:46.078302 11932 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.881594 13939 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.081907 12041 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:58:46.040418 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 192154us
I20250113 08:58:46.084705 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.085484 12007 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.086884 13917 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.087275 13962 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.088794 11932 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:45.979755 12038 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
W20250113 08:58:46.040076 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 203174us
W20250113 08:58:45.976470 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 115651us
I20250113 08:58:46.088549 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:45.992086 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:46.098726 13897 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:45.974314 12040 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:46.100021 12724 raft_consensus.cc:1235] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:46.100347 11932 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.101696 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.102218 13883 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:46.102586 13883 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:46.102936 13883 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:46.101003 13897 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.117704 11932 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:45.908094 13885 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:46.058415 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:45.894878 13946 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:45.978509 13953 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.050134 13901 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:46.120702 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.121733 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.122720 11931 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.124009 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:46.124365 12041 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:46.125418 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.051318 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:46.056049 13910 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:46.217028 11934 consensus_peers.cc:487] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc84bb347ec2413e9188b7253b0812ab. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:46.125691 11932 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.171238 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.126317 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.152172 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.219105 13957 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.123984 13926 consensus_queue.cc:1046] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20250113 08:58:46.176023 13961 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:46.221611 13961 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:46.221947 13926 raft_consensus.cc:3049] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.171470 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.222370 13926 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:46.172183 13939 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.222838 12684 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.222126 13961 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.125975 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:58:46.154073 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 162081us
I20250113 08:58:46.226943 11858 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.227417 13939 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.227918 13963 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.224615 12684 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.119978 13962 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.153421 13898 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.230492 13939 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.224593 13936 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.152676 11856 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:45.988122 13956 raft_consensus.cc:3049] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.236383 13956 raft_consensus.cc:738] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:46.237288 13956 consensus_queue.cc:260] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.238518 13956 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.230974 12684 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:58:46.218761 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 93163.6us
I20250113 08:58:46.245158 11932 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:58:46.245738 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 93283.3us
I20250113 08:58:46.246184 13962 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.246516 11856 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.247304 13928 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.247678 11856 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.223224 13926 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:46.232270 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 50753.6us
I20250113 08:58:46.257972 12007 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.258080 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.258819 13958 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.259421 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.260835 11890 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:46.262420 13963 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.262665 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.263123 13964 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.263978 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.122843 13912 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:46.235755 12038 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:46.264601 11856 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.121874 13923 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.265390 13928 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.229697 11858 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.256450 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.265857 11856 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.265345 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.176734 12006 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.267138 12007 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.267572 11858 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:58:46.267843 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 91214.5us
I20250113 08:58:46.235141 13910 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.268627 12006 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.268839 11858 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.269554 13949 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.269620 13950 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.229082 13883 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.270325 11962 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:46.270686 13963 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.271076 12721 raft_consensus.cc:1235] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:46.274585 13949 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.271068 13964 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.257210 13936 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.275750 11967 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:46.276270 13883 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.277500 12007 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.278327 13950 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.278509 13943 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.278936 13950 consensus_queue.cc:1046] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:46.279979 13958 raft_consensus.cc:3049] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.280200 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.280350 13958 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:46.281299 13958 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.281816 13950 consensus_queue.cc:1046] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:46.282796 13933 raft_consensus.cc:3049] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.283164 13933 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:46.283346 13950 consensus_queue.cc:1046] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:58:46.284394 13943 raft_consensus.cc:3049] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:46.283943 13933 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.284775 13943 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:46.283931 11856 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.286533 12006 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.285533 13933 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.285681 13943 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.289619 13943 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:46.295245 11856 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.296504 11856 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.297628 13949 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.299124 12684 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.299279 13958 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.300632 12684 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.301838 11856 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.304870 11932 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.304659 13928 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.308467 13939 consensus_queue.cc:1046] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:46.309283 13942 consensus_queue.cc:1046] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:46.310456 13883 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.310688 13942 raft_consensus.cc:3049] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.311100 13942 raft_consensus.cc:738] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.311904 11932 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.312451 12684 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.311157 13883 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250113 08:58:46.313710 13942 consensus_queue.cc:260] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.314836 13959 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.315238 13939 raft_consensus.cc:3049] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.313043 11932 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.315685 13939 raft_consensus.cc:738] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:46.316053 13942 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.316450 13883 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.318060 12684 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.318905 11856 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.319799 12006 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.321072 12006 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.322782 13928 consensus_queue.cc:1046] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:46.323256 11964 raft_consensus.cc:1235] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:46.323447 11962 raft_consensus.cc:1235] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:46.323902 13928 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.324575 13928 raft_consensus.cc:3049] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.324891 13928 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.305027 13955 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.325069 13897 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:58:46.319528 13966 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.325949 13925 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.311723 13949 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.326303 13908 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.315232 13936 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:46.335557 13936 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:46.335971 13936 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.316576 13939 consensus_queue.cc:260] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.304158 13950 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.337872 12684 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.326973 11856 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.333215 12006 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.333878 13901 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.333997 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:46.341038 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 218421us
I20250113 08:58:46.341773 11931 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.340322 13949 consensus_queue.cc:1046] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250113 08:58:46.342682 13939 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.322129 13918 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.346660 13928 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.348050 13928 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.349411 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.350066 13949 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.350509 13966 raft_consensus.cc:3049] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.338799 13897 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:58:46.352101 13964 consensus_queue.cc:1046] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:46.356089 13966 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.356812 13964 raft_consensus.cc:3049] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.357291 13964 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.356977 13966 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.355340 13936 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.358400 13966 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.358248 13964 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.360247 13956 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.360472 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.353044 13961 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.362293 11961 raft_consensus.cc:1235] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:46.362931 13961 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.364801 11931 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.352675 13926 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.369318 13950 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.370035 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.370736 13910 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.371477 13910 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.371665 12684 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.373042 12684 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.373172 13955 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.375250 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.375541 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:46.375543 13897 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:46.376927 13964 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.378072 12006 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.378856 11856 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.379967 11856 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.381431 12006 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.384079 12006 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.383210 13897 raft_consensus.cc:695] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:46.385833 13897 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.384133 13943 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.392918 11931 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.393085 12684 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.393592 12006 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.394938 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.395586 12006 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.395789 11931 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:58:46.395902 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 45460.3us
I20250113 08:58:46.396329 12684 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.397944 11931 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.398535 12006 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.399955 11931 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.400060 12006 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.400693 13923 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:46.401427 13923 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:46.401556 13972 raft_consensus.cc:2743] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:58:46.380721 13949 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.403024 12006 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.403080 11931 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.401983 13923 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:46.394358 13943 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.402329 13898 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.406721 11931 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.394832 13950 raft_consensus.cc:2743] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.410246 13962 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.410938 13962 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:46.402462 13959 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:46.411303 13962 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:46.411581 13959 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:46.411939 13962 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.412084 13959 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.411186 11931 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.408489 12006 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.396306 13925 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.399199 13901 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.404126 13908 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.426827 13901 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.406070 13946 raft_consensus.cc:2743] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.429435 13898 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:46.429656 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.429931 13898 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:46.430320 13898 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:46.398276 13936 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.399410 13918 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.434264 11788 catalog_manager.cc:5526] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 4 to 5, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 5 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:58:46.423205 11931 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.420153 12006 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.418844 13972 raft_consensus.cc:2743] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.398617 13968 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:46.447376 12006 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.450554 13968 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:46.413621 13917 raft_consensus.cc:2743] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.413367 13882 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.452085 11931 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.453364 11931 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.451038 13968 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.454643 13912 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.455080 13902 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.455300 13883 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.455819 13961 consensus_queue.cc:1046] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:46.456980 13961 raft_consensus.cc:3049] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.457338 13961 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:46.458243 13961 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.459641 13961 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.461424 13926 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.463075 11891 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250113 08:58:46.455132 11931 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.466899 13902 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250113 08:58:46.465955 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68801.2us
I20250113 08:58:46.467731 11931 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.468389 13917 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.465469 13962 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.464998 13912 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.471642 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.472975 12686 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.474045 13962 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:46.474198 12686 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.474843 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.475329 12686 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.475550 13939 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.476228 13955 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.475922 13898 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.476630 11961 raft_consensus.cc:1235] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:58:46.476773 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:46.477305 13955 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.478787 12686 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:58:46.480453 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 100554us
I20250113 08:58:46.481429 11856 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.482776 12684 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.479213 13917 consensus_queue.cc:1046] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:46.484958 12037 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250113 08:58:46.484866 13959 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.486274 13939 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.487107 13959 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.479776 13902 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.465132 13956 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20250113 08:58:46.489921 13949 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:46.490325 13949 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:46.490722 13949 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.496055 12725 raft_consensus.cc:1235] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:46.469334 13966 consensus_queue.cc:1046] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:46.498133 11856 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.501235 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.502169 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.503129 13917 consensus_queue.cc:1046] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:46.503475 13946 raft_consensus.cc:3049] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:58:46.503881 13946 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:46.504142 13917 raft_consensus.cc:3049] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.504498 13917 raft_consensus.cc:738] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:46.505805 12042 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:46.505734 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.506592 13898 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.507556 13917 consensus_queue.cc:260] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.509003 13917 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.510821 12038 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:46.492628 13882 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250113 08:58:46.516857 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 114132us
I20250113 08:58:46.517189 12723 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:46.519327 11856 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.504688 13946 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.523351 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.530812 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.536898 12006 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.531162 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.567503 13933 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 08:58:46.571709 13910 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.571338 13968 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.520548 13928 raft_consensus.cc:3049] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.580062 13928 raft_consensus.cc:738] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.581058 13928 consensus_queue.cc:260] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:46.585621 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 51606us
W20250113 08:58:46.587050 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 63817.4us
I20250113 08:58:46.587577 13964 raft_consensus.cc:2743] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.587832 11856 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:46.590273 13963 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:46.590700 13963 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:46.591043 13963 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.514978 13923 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:46.597496 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 66924.9us
I20250113 08:58:46.598821 11961 raft_consensus.cc:1235] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:46.588972 11856 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.574859 13968 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:46.522109 13973 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.631647 13966 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250113 08:58:46.623005 13957 raft_consensus.cc:2949] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
W20250113 08:58:46.624357 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 62619.6us
I20250113 08:58:46.651775 12039 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:46.612877 13964 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:58:46.523110 13970 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.644102 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:46.615974 13928 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.639873 11886 raft_consensus.cc:2949] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:46.632989 13966 consensus_queue.cc:1046] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250113 08:58:46.656962 13972 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:58:46.661348 13962 consensus_queue.cc:1046] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20250113 08:58:46.658785 13923 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.658006 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:46.693276 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 59211.7us
I20250113 08:58:46.671377 13972 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.707019 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.707592 13972 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.702392 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.687063 13883 raft_consensus.cc:988] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:46.713310 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.702859 11967 raft_consensus.cc:1235] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:46.703169 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.703365 11964 raft_consensus.cc:1235] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:46.703728 13885 raft_consensus.cc:3049] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:46.715104 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.716127 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.702915 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.704970 11962 raft_consensus.cc:1235] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:46.705142 11966 raft_consensus.cc:1235] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:46.718963 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.720490 12039 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250113 08:58:46.724826 12686 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.729590 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.730211 13881 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250113 08:58:46.727742 13882 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.737756 11931 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.739820 11931 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.738916 13968 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250113 08:58:46.715044 13885 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:46.742496 13979 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.736806 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.742262 13885 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.728169 13883 raft_consensus.cc:1076] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:46.750195 13885 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:46.757036 13910 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:46.757733 13910 raft_consensus.cc:695] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:46.760047 11931 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.759541 13910 consensus_queue.cc:237] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.761412 13898 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:46.761487 11961 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:46.761936 13898 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:46.762919 13898 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.738161 13912 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:58:46.766721 11931 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.736461 13980 consensus_queue.cc:1046] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:46.728868 13972 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:58:46.695086 13949 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.773707 13912 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.775898 11931 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.770947 11886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:46.781630 11931 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.776902 13957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:54914 (request call id 10) took 1039 ms. Trace:
I20250113 08:58:46.779808 11886 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:46.785494 11886 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.781780 11961 raft_consensus.cc:1235] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:46.782912 11931 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.787038 13898 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.778966 13942 raft_consensus.cc:3049] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:46.789309 11931 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.795313 12686 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.797766 13898 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:58:46.782270 13946 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:46.782155 13979 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.785039 13912 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:46.812294 11959 raft_consensus.cc:1235] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:46.810285 11886 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.790675 13942 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.819686 11886 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.819497 13942 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.785387 13957 rpcz_store.cc:276] 0113 08:58:45.736993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:45.737092 (+ 99us) service_pool.cc:225] Handling call
0113 08:58:45.778137 (+ 41045us) log.cc:844] Serialized 513 byte log entry
0113 08:58:46.081620 (+303483us) spinlock_profiling.cc:243] Waited 9.06 ms on lock 0x7b18002b5be0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc33364893a 00007fc33cb04be9 00007fc333509ad0 00007fc3336ce304 00007fc3336cdbf5 00007fc336583580 00007fc3365e9113 00007fc33660f7e2 00007fc33b568f29
0113 08:58:46.093773 (+ 12153us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b18002b5be0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d10d8 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:58:46.098809 (+ 5036us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b18002b5be0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d1204 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:58:46.211073 (+112264us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b18002b5be0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc33369e7b6 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33659acaf 00007fc3365837a5 00007fc3365889d1
0113 08:58:46.211667 (+ 594us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b540021aaa0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33659acaf 00007fc3365837a5 00007fc3365889d1 00007fc336588959 00007fc3365888e9 00007fc3365888b1 00007fc336587a7d 00007fc33c70d234 00007fc33c70d069 00007fc3336d1156
0113 08:58:46.776797 (+565130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34308,"raft.queue_time_us":90364,"raft.run_cpu_time_us":12339,"raft.run_wall_time_us":361662,"spinlock_wait_cycles":215618176,"thread_start_us":95541,"threads_started":2,"wal-append.queue_time_us":261728}
I20250113 08:58:46.808348 13936 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.843492 11886 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:46.844246 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.846918 12686 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:46.821007 13942 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:46.809799 13918 consensus_queue.cc:1046] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20250113 08:58:46.846786 13948 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.856117 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:46.809798 13964 consensus_queue.cc:1046] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250113 08:58:46.858552 13964 raft_consensus.cc:3049] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.859084 13964 raft_consensus.cc:738] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.831074 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.861948 11959 raft_consensus.cc:3049] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.862263 11959 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:46.862316 13918 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:46.862993 13918 raft_consensus.cc:695] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:46.863149 11959 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.864578 11959 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.864136 13918 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.825381 13959 raft_consensus.cc:3049] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.831058 13949 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:46.818167 13966 consensus_queue.cc:1046] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20250113 08:58:46.865483 13953 raft_consensus.cc:3049] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:46.834455 13956 consensus_queue.cc:1046] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20250113 08:58:46.860311 13964 consensus_queue.cc:260] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.883149 13959 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:46.898716 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.899510 12035 raft_consensus.cc:2388] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 2: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:46.886956 13953 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:46.899044 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:46.902231 12033 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.877393 13948 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20250113 08:58:46.910741 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:46.924077 13971 raft_consensus.cc:3049] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:46.924553 13971 raft_consensus.cc:738] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:46.925401 13971 consensus_queue.cc:260] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.926818 13971 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:46.892199 13963 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.931757 12041 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:58:46.932670 13963 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:46.934742 11893 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:46.934302 13953 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.935899 13953 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.935735 11787 catalog_manager.cc:5526] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:46.938879 11856 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:46.937115 13979 consensus_queue.cc:1046] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:46.939985 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:46.940526 13979 raft_consensus.cc:3049] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:46.940973 13979 raft_consensus.cc:738] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:46.941910 13979 consensus_queue.cc:260] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:46.943490 13979 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:46.946065 13948 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:46.927924 13964 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:46.949119 13959 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:46.891605 12042 raft_consensus.cc:2949] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:46.969513 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:58:46.973964 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 88564us
I20250113 08:58:46.981894 11856 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:46.964931 13970 raft_consensus.cc:2743] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:46.988801 13966 raft_consensus.cc:2743] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:47.022395 12033 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:47.024605 12041 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:47.025220 11856 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:47.026209 12721 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:47.027729 13959 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:47.027909 13988 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:47.028481 13988 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:47.028250 13970 consensus_queue.cc:1046] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:47.029695 13970 raft_consensus.cc:3049] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:47.029471 13988 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:47.030126 13970 raft_consensus.cc:738] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:47.052598 11961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:47.053350 11961 raft_consensus.cc:604] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:47.062956 13989 ts_tablet_manager.cc:927] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:47.068328 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250113 08:58:47.070680 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:47.071429 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:47.052914 13970 consensus_queue.cc:260] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:47.193457 13990 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
I20250113 08:58:47.168013 13955 raft_consensus.cc:3049] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:47.211576 13970 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:47.201563 13955 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:58:47.228830 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 75373us
I20250113 08:58:47.246377 13957 raft_consensus.cc:988] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:47.247051 13957 raft_consensus.cc:1076] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:47.253199 11891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:47.253835 11891 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:47.254182 11891 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:47.225891 13989 tablet_copy_client.cc:323] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.10.115.130:41863
I20250113 08:58:47.281414 12041 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:58:47.285588 11977 tablet_copy_service.cc:140] P 37bd4bed3ee6423c8dac0cef3bf6882a: Received BeginTabletCopySession request for tablet fc84bb347ec2413e9188b7253b0812ab from peer 5f9c33777f6845c2848a2a6102a1c2ff ({username='slave'} at 127.0.0.1:54924)
I20250113 08:58:47.286243 11893 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:47.286402 11977 tablet_copy_service.cc:161] P 37bd4bed3ee6423c8dac0cef3bf6882a: Beginning new tablet copy session on tablet fc84bb347ec2413e9188b7253b0812ab from peer 5f9c33777f6845c2848a2a6102a1c2ff at {username='slave'} at 127.0.0.1:54924: session id = 5f9c33777f6845c2848a2a6102a1c2ff-fc84bb347ec2413e9188b7253b0812ab
I20250113 08:58:47.266172 13955 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:47.286649 13912 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:47.288484 11891 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:47.338403 11891 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:47.342263 11893 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:58:47.347766 11959 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:58:47.348464 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 59267.6us
I20250113 08:58:47.351352 11891 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:47.357730 12035 raft_consensus.cc:1270] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:47.358671 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:47.359282 12041 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:47.363819 13979 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:47.370307 11889 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:47.371243 13955 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:47.387894 12041 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:47.390127 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:47.390947 11961 raft_consensus.cc:3049] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:47.391301 11961 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:47.399490 11856 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:47.392228 11961 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:47.408725 11961 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:47.409235 11891 raft_consensus.cc:1270] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:58:47.408003 13959 consensus_queue.cc:1035] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20250113 08:58:47.417572 13980 consensus_queue.cc:1035] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:58:47.413239 13942 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:47.419920 13942 raft_consensus.cc:695] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:47.433990 11890 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:47.446010 11890 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:58:47.447409 13882 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:58:47.453923 11788 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44470 (request call id 244) took 1051 ms. Trace:
I20250113 08:58:47.430719 13942 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:47.454397 11788 rpcz_store.cc:276] 0113 08:58:46.402703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:46.402953 (+ 250us) service_pool.cc:225] Handling call
0113 08:58:47.453840 (+1050887us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:58:46.579617 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:58:46.580674 (+ 1057us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:58:46.580785 (+ 111us) write_op.cc:435] Acquired schema lock
0113 08:58:46.580824 (+ 39us) tablet.cc:659] Decoding operations
0113 08:58:46.581416 (+ 592us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:58:46.581535 (+ 119us) write_op.cc:641] Partition lock acquired for write op
0113 08:58:46.581575 (+ 40us) tablet.cc:682] Acquiring locks for 1 operations
0113 08:58:46.617675 (+ 36100us) tablet.cc:698] Row locks acquired
0113 08:58:46.626472 (+ 8797us) write_op.cc:260] PREPARE: finished
0113 08:58:46.660555 (+ 34083us) write_op.cc:270] Start()
0113 08:58:46.668486 (+ 7931us) write_op.cc:276] Timestamp: P: 1736758726655150 usec, L: 0
0113 08:58:46.669900 (+ 1414us) op_driver.cc:348] REPLICATION: starting
0113 08:58:46.670583 (+ 683us) log.cc:844] Serialized 539 byte log entry
0113 08:58:47.179813 (+509230us) op_driver.cc:464] REPLICATION: finished
0113 08:58:47.252380 (+ 72567us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b1800391f60. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d10d8 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:58:47.252574 (+ 194us) write_op.cc:301] APPLY: starting
0113 08:58:47.252705 (+ 131us) tablet.cc:1364] starting BulkCheckPresence
0113 08:58:47.252862 (+ 157us) tablet.cc:1367] finished BulkCheckPresence
0113 08:58:47.252888 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:58:47.255762 (+ 2874us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:58:47.256037 (+ 275us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0113 08:58:47.256094 (+ 57us) write_op.cc:312] APPLY: finished
0113 08:58:47.256469 (+ 375us) log.cc:844] Serialized 23 byte log entry
0113 08:58:47.256708 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:58:47.259894 (+ 3186us) write_op.cc:454] Released schema lock
0113 08:58:47.265564 (+ 5670us) write_op.cc:341] FINISH: Updating metrics
0113 08:58:47.367879 (+102315us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:58:47.367920 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":41884,"apply.run_cpu_time_us":2482,"apply.run_wall_time_us":90095,"num_ops":1,"prepare.queue_time_us":133989,"prepare.run_cpu_time_us":30478,"prepare.run_wall_time_us":294561,"raft.queue_time_us":281458,"raft.run_cpu_time_us":1906,"raft.run_wall_time_us":38352,"replication_time_us":509706,"spinlock_wait_cycles":9114880,"thread_start_us":541714,"threads_started":4,"wal-append.queue_time_us":225608}]]}
I20250113 08:58:47.683409 11961 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:47.774672 13957 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:47.775252 13936 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:47.775381 13957 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:47.775960 13936 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:47.777990 13957 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:47.413125 13979 consensus_queue.cc:1159] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6ea00811b7d44b49b872ac5c03b91487" responder_term: 3 status { last_received { term: 2 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250113 08:58:47.800385 13970 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:47.801306 13973 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:47.799785 13964 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:47.860598 11839 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 6ea00811b7d44b49b872ac5c03b91487 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } }
I20250113 08:58:47.868580 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:47.855834 13973 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:47.852432 13970 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:47.858211 13964 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:47.896106 11890 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:47.892249 13936 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:47.893607 14011 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:47.931543 14005 log.cc:926] Time spent T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20250113 08:58:47.919045 14011 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:47.962100 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 163796us
I20250113 08:58:47.965886 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:47.966800 11968 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:47.967916 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:47.976141 11959 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250113 08:58:47.981848 14006 log.cc:926] Time spent T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20250113 08:58:47.983103 14011 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:47.963316 14012 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:47.984665 13973 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:47.985240 14012 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:47.997438 14007 log.cc:926] Time spent T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20250113 08:58:47.985702 14017 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:47.998832 11967 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:48.053810 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:48.056324 12033 raft_consensus.cc:2388] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:47.990617 13970 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.060425 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:47.992339 13964 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:47.992803 14012 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.062212 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:48.077517 12726 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:48.043470 14011 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.052124 13926 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.051862 13939 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.051717 13883 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:48.054342 13917 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.052289 13936 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:48.061316 13971 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.095494 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:48.059970 13956 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:48.067325 13882 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:48.074077 13973 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.103782 11886 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:48.111617 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:48.068213 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:47.998374 14017 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.122817 11858 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:48.087075 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:48.138386 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:48.095111 13970 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:48.139220 12041 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:48.114346 13957 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.179903 12721 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:48.178097 14014 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.178550 13950 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.090183 14011 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.137741 12036 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:58:48.097201 13926 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.206290 11934 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:48.188616 13988 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:48.210434 13897 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.210567 12042 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:48.111800 13971 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.130906 13882 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.197759 14014 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.181949 13918 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.205390 13957 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.187552 14020 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.137435 11959 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:48.122416 13956 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.223665 14017 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.107170 13936 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.193817 14013 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250113 08:58:48.107432 13917 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.131173 13973 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.104043 13883 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.094874 13939 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.198164 13950 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:48.233378 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 116839us
I20250113 08:58:48.236930 13897 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.533792 14011 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.534395 13883 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.535043 13957 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.535718 13918 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.537107 13950 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.221042 14032 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.537662 13926 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.299547 13977 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:48.340821 13964 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.538604 13977 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.539059 13964 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.540036 13882 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:48.537909 14032 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.541421 14020 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.302925 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:48.543494 12719 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:48.543802 14020 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.304235 13961 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:48.546049 13961 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:48.547420 13961 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:48.548552 13917 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.549575 12723 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:48.231169 14012 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.546393 14032 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.537879 13918 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:48.552245 13918 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.551965 14012 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.546613 13936 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:48.552767 13918 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:48.458642 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 370379us
I20250113 08:58:48.565692 13956 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:48.573758 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:48.544507 13897 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:48.574753 12723 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:48.304601 11790 catalog_manager.cc:5526] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 3 to 4, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 4 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:48.232237 13988 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:48.600555 11859 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:48.420970 11977 tablet_copy_source_session.cc:215] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:58:48.445075 13881 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.310711 14024 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:48.258385 14025 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:48.606900 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 75520.7us
I20250113 08:58:48.540619 13977 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:48.596294 11790 catalog_manager.cc:5526] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: config changed from index -1 to 3, term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to <none>, NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) added. New cstate: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } } }
I20250113 08:58:48.203105 13970 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:48.619863 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 77213.4us
I20250113 08:58:48.620285 11932 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:48.599753 13958 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:48.621495 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 494929us
W20250113 08:58:48.630103 13953 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.009s sys 0.018s
W20250113 08:58:48.649804 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.510s user 0.009s sys 0.015s
I20250113 08:58:48.664448 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:48.664695 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 525042us
I20250113 08:58:48.604224 14026 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.607515 13988 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:48.600768 14011 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:48.678858 12723 raft_consensus.cc:2388] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:58:48.704465 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 84290.5us
I20250113 08:58:48.705487 13957 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:48.706091 13957 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:48.706540 13957 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:48.708833 12007 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:48.618808 13917 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:48.709872 13917 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:48.710681 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45302.6us
I20250113 08:58:48.634775 13882 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:48.678931 13953 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.744s user 0.009s sys 0.018s
I20250113 08:58:48.651703 14017 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.615319 14020 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.718713 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:48.651341 13939 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:48.754750 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68889.9us
I20250113 08:58:48.651464 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:48.753576 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 44848.8us
I20250113 08:58:48.766709 11968 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:48.776038 11858 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:48.730158 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:48.717159 14011 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.776947 11966 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:48.622639 13881 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:48.804569 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 204081us
I20250113 08:58:48.805132 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:48.680039 13958 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.773834 14012 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.790524 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:48.653204 14025 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.763034 13882 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.636824 14024 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:48.856711 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 50139.4us
I20250113 08:58:48.792761 12719 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:48.794916 13918 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:48.704010 14026 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.826470 11977 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:54924 (request call id 321) took 1542 ms. Trace:
I20250113 08:58:48.773422 14017 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.855206 14025 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:48.854823 13882 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:48.881676 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.014s sys 0.028s
W20250113 08:58:48.950084 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.841s user 0.015s sys 0.028s
I20250113 08:58:48.951184 14014 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:48.951463 13942 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:48.951866 13942 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:48.952220 13942 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:48.952296 14014 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:48.952536 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 94675.7us
I20250113 08:58:48.952768 14014 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.953354 14046 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:48.953898 14046 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.955021 13970 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.955554 13970 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:48.955739 13980 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.956359 13980 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:48.858219 12008 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.000s sys 0.044s
W20250113 08:58:48.957438 12008 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.000s sys 0.045s
W20250113 08:58:48.957757 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 876697us
I20250113 08:58:48.904641 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:48.958423 14038 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.958891 12037 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:48.904832 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:48.959491 12035 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:58:48.927809 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68961.1us
I20250113 08:58:48.961275 12007 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:48.779548 14020 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:48.955826 14046 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:48.956789 13971 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:48.859982 13948 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.016s sys 0.026s
I20250113 08:58:48.945506 13959 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.793041 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:48.947970 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:48.767699 14031 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:48.975615 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:48.975350 13912 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.450069 11891 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:48.857084 14012 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:48.982614 12007 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:48.977568 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:48.854864 13958 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:48.802423 13973 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:49.049933 14016 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:48.975278 13979 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.004294 14040 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:48.975278 13883 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.005996 14042 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:48.927528 13977 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:48.959141 13968 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:48.955883 13970 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:48.958941 14038 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:48.928833 14027 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.358031 11856 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:49.192349 11790 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.511s user 0.007s sys 0.016s
I20250113 08:58:49.449594 11961 raft_consensus.cc:2388] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:58:49.003790 13948 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.777s user 0.016s sys 0.026s
I20250113 08:58:49.346809 14046 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.146291 13917 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:49.437812 13950 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:58:49.103634 13961 consensus_meta.cc:287] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.555s user 0.006s sys 0.006s
I20250113 08:58:49.453701 12685 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:49.447715 14034 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.449797 14031 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:49.110713 13957 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:49.141595 14026 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:49.026360 14044 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:49.316987 14029 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.012s sys 0.019s
I20250113 08:58:49.454408 14012 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:49.454708 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:49.449740 14032 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.447180 13959 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:49.455559 12717 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:49.456269 13980 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:49.448601 14020 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:49.458214 13882 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.458680 13918 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:49.459565 14036 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.462651 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:49.139674 13881 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:49.746012 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.000s sys 0.011s
I20250113 08:58:49.146944 13926 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.813997 14036 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:49.814985 13950 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:49.815107 13926 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:49.812986 14021 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.809813 14056 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.570016 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:49.814627 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.814s user 0.000s sys 0.013s
I20250113 08:58:49.154167 14051 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.818079 13926 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:49.593621 11858 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.810s user 0.013s sys 0.019s
I20250113 08:58:49.817029 14036 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:50.141178 12008 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.008s sys 0.012s
I20250113 08:58:49.098479 14019 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:48.926692 11977 rpcz_store.cc:276] 0113 08:58:47.284462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:47.284961 (+ 499us) service_pool.cc:225] Handling call
0113 08:58:48.363564 (+1078603us) spinlock_profiling.cc:243] Waited 92.4 ms on lock 0x7b540021aaa0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336599990 00007fc3366133f5 00007fc33661333e 00007fc33b543436 00007fc33b54ce3e 00007fc332fed62b 00007fc33b54cdba 00007fc33b547ecc 00007fc33b54307b 00007fc33b537c35
0113 08:58:48.418615 (+ 55051us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b5400196754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336613347 00007fc33b543436 00007fc33b54ce3e 00007fc332fed62b 00007fc33b54cdba 00007fc33b547ecc 00007fc33b54307b 00007fc33b537c35 00007fc33a62ca74 00007fc33a62ca02
0113 08:58:48.826362 (+407747us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1021502592}
I20250113 08:58:49.890924 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:49.324752 13964 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:49.144415 14011 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:49.349061 14013 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.555554 13973 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.468429 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:50.167436 12725 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:49.145901 14041 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:49.868922 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:49.948762 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:49.812332 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:49.820065 13942 consensus_meta.cc:287] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.867s user 0.011s sys 0.000s
I20250113 08:58:50.105381 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 390) took 1202 ms. Trace:
I20250113 08:58:49.886541 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:49.651074 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:49.684567 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:50.154992 12722 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
W20250113 08:58:50.152837 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.376s user 0.022s sys 0.027s
W20250113 08:58:50.153635 12008 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.701s user 0.008s sys 0.012s
I20250113 08:58:49.635565 11932 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:49.870286 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:50.104157 12006 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.007s sys 0.010s
I20250113 08:58:49.813477 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:49.892295 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:49.919250 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:50.165377 11890 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.007s sys 0.009s
I20250113 08:58:50.166157 12726 raft_consensus.cc:2388] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:49.939668 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:50.156181 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 798222us
W20250113 08:58:49.465420 13988 consensus_meta.cc:287] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.749s user 0.012s sys 0.001s
I20250113 08:58:50.166023 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 1991) took 1437 ms. Trace:
I20250113 08:58:50.171505 11967 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:49.750427 14015 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:50.157083 13970 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:50.119885 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 391) took 1210 ms. Trace:
I20250113 08:58:50.217907 12039 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:50.217842 13942 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:50.217780 12006 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.007s sys 0.018s
I20250113 08:58:50.219331 11893 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:49.552558 14014 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.220265 12724 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:50.159636 14062 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:50.174909 12723 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:49.143548 14024 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:50.175343 11884 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
W20250113 08:58:50.164548 12007 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.007s sys 0.018s
I20250113 08:58:50.260632 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:49.923203 13926 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.514533 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:49.921514 13881 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:49.650144 14035 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:49.632476 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:50.262696 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 627228us
I20250113 08:58:49.479597 13958 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:49.697324 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:49.631758 14022 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:49.650063 13936 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:58:50.260164 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.4841e+06us
I20250113 08:58:50.167829 14032 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:50.262482 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.286s user 0.000s sys 0.034s
W20250113 08:58:50.330096 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.354s user 0.000s sys 0.035s
I20250113 08:58:50.221792 11891 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:58:50.261013 11892 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:50.208982 13973 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.160481 13917 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:49.772543 14049 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:49.653337 14030 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:50.103642 13953 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.156023 14042 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:49.817131 13950 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:49.665673 14017 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:50.152864 14029 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.359s user 0.027s sys 0.028s
I20250113 08:58:50.271651 11961 rpcz_store.cc:276] 0113 08:58:48.718588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.720635 (+ 2047us) service_pool.cc:225] Handling call
0113 08:58:50.155837 (+1435202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:49.774204 14052 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:50.219624 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:50.277667 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 1988) took 1565 ms (client timeout 1839 ms). Trace:
I20250113 08:58:50.161031 14031 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:50.159668 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 464) took 1502 ms (client timeout 1511 ms). Trace:
I20250113 08:58:50.284361 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1534) took 1363 ms. Trace:
W20250113 08:58:50.288031 11859 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.008s sys 0.015s
W20250113 08:58:50.289069 12007 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.008s sys 0.022s
I20250113 08:58:49.948105 14061 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:50.171473 13980 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:49.714262 14025 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:50.288729 12725 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:50.166944 12719 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:49.922727 13968 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:49.426590 14050 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.845s
I20250113 08:58:50.260164 14036 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:49.922749 14020 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:50.220099 12042 rpcz_store.cc:276] 0113 08:58:48.902526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.902674 (+ 148us) service_pool.cc:225] Handling call
0113 08:58:50.104327 (+1201653us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:58:50.104619 (+ 292us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.104881 (+ 262us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:50.104918 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:50.104975 (+ 57us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:50.105018 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:50.105094 (+ 76us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:50.105271 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49739648}
I20250113 08:58:50.159965 13882 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:49.946139 14057 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:50.174860 14011 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:49.515154 13912 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:49.871876 14060 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:50.160880 13883 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.415316 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:50.393369 13957 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.416041 12042 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:50.416443 13957 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.417318 13883 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:50.417032 13957 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.378859 14049 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.417997 13883 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.418697 13883 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.419875 14049 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:50.269343 13923 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:50.172255 14013 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.303812 14024 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:50.312804 14069 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:50.160629 14046 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.160209 13918 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.288316 13964 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.161458 13977 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:50.315217 11934 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.511s (SENT) [suppressed 8 similar messages]
W20250113 08:58:50.261147 11890 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s user 0.007s sys 0.017s
I20250113 08:58:50.156658 13979 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.160280 13961 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.270269 13988 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.321192 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:50.267535 12041 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:58:50.324517 14026 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.331207 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:49.889302 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:49.493829 14016 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:50.342216 12005 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.013s sys 0.009s
W20250113 08:58:50.342687 11933 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.014s sys 0.020s
I20250113 08:58:49.814005 14044 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:50.329473 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 203539us
I20250113 08:58:49.922806 14051 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.320856 14068 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:50.160075 14040 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.216936 14063 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:50.310599 12721 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:50.165081 14027 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.341462 14065 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:50.287631 14062 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.355592 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:50.175477 14021 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.168298 14019 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:50.357131 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 903481us
I20250113 08:58:50.175696 14056 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.259879 14041 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.169622 14034 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.421538 14012 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.567365 13968 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.432446 13942 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.447547 14038 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:50.563140 14039 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:50.397671 14020 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.571509 14034 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:50.304119 13926 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:50.383116 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.40047e+06us
W20250113 08:58:50.384552 11859 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.702s user 0.009s sys 0.022s
W20250113 08:58:50.572902 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 932835us
W20250113 08:58:50.573767 12005 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.997s user 0.014s sys 0.013s
I20250113 08:58:50.336640 12718 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:50.381774 12039 rpcz_store.cc:276] 0113 08:58:48.908791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.908918 (+ 127us) service_pool.cc:225] Handling call
0113 08:58:49.886308 (+977390us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:50.271974 (+385666us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49450240}
W20250113 08:58:50.382040 11967 rpcz_store.cc:269] 0113 08:58:48.712097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.752291 (+ 40194us) service_pool.cc:225] Handling call
0113 08:58:48.977391 (+225100us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b58001c4198. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:50.277503 (+1300112us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54000514d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:50.277610 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1673648896}
I20250113 08:58:50.311547 13881 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.350028 13917 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.374246 14042 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:50.329074 13958 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.387600 14029 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:50.271333 12036 rpcz_store.cc:276] 0113 08:58:48.902915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.927248 (+ 24333us) service_pool.cc:225] Handling call
0113 08:58:49.890203 (+962955us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:58:49.891727 (+ 1524us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:49.938501 (+ 46774us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:49.946960 (+ 8459us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:49.956806 (+ 9846us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:49.961835 (+ 5029us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:50.104237 (+142402us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:50.112925 (+ 8688us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51038592}
I20250113 08:58:50.336434 13936 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.398344 13959 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:50.399957 12008 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:50.379652 13950 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:50.380056 14022 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.391111 14025 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.414665 14066 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.411237 14011 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.281131 14015 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:50.417140 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:50.433522 14060 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:50.335882 14035 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.437623 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:50.383162 14030 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.442224 14024 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.443408 13882 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:50.378473 14064 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:50.420588 14080 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:50.376017 13953 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.388692 13980 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.571805 14039 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.576944 14017 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:50.577127 14020 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:50.382668 12717 rpcz_store.cc:269] 0113 08:58:48.657208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.681227 (+ 24019us) service_pool.cc:225] Handling call
0113 08:58:50.159572 (+1478345us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:50.392843 14061 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.421242 14075 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:50.848731 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:50.820363 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:50.869680 11968 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:50.869779 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 399) took 1432 ms. Trace:
I20250113 08:58:50.865226 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:50.871555 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 392) took 1968 ms (client timeout 2532 ms). Trace:
I20250113 08:58:50.872558 11892 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:50.873600 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:50.537638 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 106927us
I20250113 08:58:50.870733 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:50.429944 13923 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:50.870576 12726 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:50.875666 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 704) took 1235 ms. Trace:
W20250113 08:58:50.702739 11933 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.275s user 0.020s sys 0.020s
I20250113 08:58:50.785683 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:50.880211 11891 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:50.787034 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 396) took 1354 ms. Trace:
I20250113 08:58:50.721135 14050 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:50.881987 14079 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:50.429446 14049 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:50.872097 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:50.870102 11963 raft_consensus.cc:2388] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:50.411690 14052 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:50.871917 12041 rpcz_store.cc:269] 0113 08:58:48.903252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.903400 (+ 148us) service_pool.cc:225] Handling call
0113 08:58:49.939445 (+1036045us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:50.871497 (+932052us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49313664}
I20250113 08:58:51.042870 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.036924 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 378) took 1546 ms. Trace:
I20250113 08:58:51.034593 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 387) took 1165 ms. Trace:
W20250113 08:58:50.912546 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 55366.7us
I20250113 08:58:50.986961 11885 raft_consensus.cc:2363] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4.
I20250113 08:58:50.568001 13912 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.956321 14075 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:50.876005 12726 rpcz_store.cc:276] 0113 08:58:49.639978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.287070 (+647092us) service_pool.cc:225] Handling call
0113 08:58:50.875605 (+588535us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:50.578280 14034 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:50.988454 11960 raft_consensus.cc:2388] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:50.885710 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.034304 13977 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:50.881567 12039 rpcz_store.cc:276] 0113 08:58:49.433060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.687416 (+1254356us) service_pool.cc:225] Handling call
0113 08:58:50.786309 (+ 98893us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.786562 (+ 253us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:50.786603 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:50.786658 (+ 55us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:50.786703 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:50.786782 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:50.786952 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:50.980310 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:50.411083 14057 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:51.065513 14049 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.882994 14079 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:51.065366 13923 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:50.886865 13964 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:51.476408 14060 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.490244 14075 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:51.485267 14035 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:51.062521 11961 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:58:51.351835 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 492160us
I20250113 08:58:51.034457 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1620) took 2165 ms. Trace:
W20250113 08:58:51.340065 11859 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.003s sys 0.019s
I20250113 08:58:51.058964 11966 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:51.097096 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:50.914417 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:51.349967 11886 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.000s sys 0.015s
I20250113 08:58:50.887014 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1619) took 2004 ms. Trace:
I20250113 08:58:51.080086 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:51.431200 12684 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.012s sys 0.005s
I20250113 08:58:50.870482 12042 rpcz_store.cc:276] 0113 08:58:49.437123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.416812 (+979689us) service_pool.cc:225] Handling call
0113 08:58:50.869648 (+452836us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:51.536068 12684 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.848s user 0.014s sys 0.005s
W20250113 08:58:51.327724 11885 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 383) took 1562 ms (client timeout 1585 ms). Trace:
I20250113 08:58:50.866222 11959 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:50.435994 13957 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.098279 13961 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.207661 13926 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.062438 11888 rpcz_store.cc:276] 0113 08:58:49.490491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.490634 (+ 143us) service_pool.cc:225] Handling call
0113 08:58:49.624468 (+133834us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.987667 (+1363199us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:50.987710 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:50.987755 (+ 45us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:50.987797 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:51.036710 (+ 48913us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540026d254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:51.036758 (+ 48us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:51.036868 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3134606592}
I20250113 08:58:51.062611 11884 rpcz_store.cc:276] 0113 08:58:49.868904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.293143 (+424239us) service_pool.cc:225] Handling call
0113 08:58:50.293544 (+ 401us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.721143 (+427599us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:50.721175 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:50.721222 (+ 47us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:50.721277 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:51.034362 (+313085us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b540038f254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:51.034401 (+ 39us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:51.034537 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":982860544}
I20250113 08:58:51.067181 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:51.538389 12041 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:58:51.537475 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 948490us
I20250113 08:58:50.954041 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:51.538336 11885 rpcz_store.cc:269] 0113 08:58:49.757124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.891924 (+134800us) service_pool.cc:225] Handling call
0113 08:58:51.314088 (+1422164us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b540034d5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:51.319010 (+ 4922us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1722775424}
I20250113 08:58:51.640034 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:51.373639 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.003s sys 0.011s
I20250113 08:58:51.703524 11966 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
W20250113 08:58:51.704005 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.126s user 0.003s sys 0.018s
I20250113 08:58:50.444384 13973 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.431643 12720 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:51.545053 11892 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:51.710211 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.710855 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1786) took 1384 ms. Trace:
I20250113 08:58:51.710978 11961 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:51.710281 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:50.858162 14038 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:51.502333 11965 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
W20250113 08:58:51.701634 11886 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s user 0.005s sys 0.021s
I20250113 08:58:51.502282 11964 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
I20250113 08:58:51.666461 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.711257 11892 rpcz_store.cc:276] 0113 08:58:50.326093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.914106 (+588013us) service_pool.cc:225] Handling call
0113 08:58:51.710713 (+796607us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:51.708031 11891 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 2: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:51.720387 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:51.720705 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1790) took 1393 ms. Trace:
I20250113 08:58:51.702435 12684 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:51.246279 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.702994 11968 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:50.869287 13971 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:50.481163 13883 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.724174 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.036875 14016 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:51.726536 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.3265e+06us
I20250113 08:58:50.687239 14012 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:51.536861 11889 rpcz_store.cc:276] 0113 08:58:48.868641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.868818 (+ 177us) service_pool.cc:225] Handling call
0113 08:58:49.427768 (+558950us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.822434 (+1394666us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:50.822474 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:50.822524 (+ 50us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:50.822564 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:51.034172 (+211608us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54000d8754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:51.034223 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:51.034359 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3206911360}
I20250113 08:58:50.886538 14032 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:51.101516 14027 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.263741 14056 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.974942 13989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc84bb347ec2413e9188b7253b0812ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:58:50.875442 14015 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:51.712579 11885 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:58:51.313424 13979 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.913812 14026 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:51.725111 11968 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
W20250113 08:58:51.704365 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.12672e+06us
I20250113 08:58:51.719887 11963 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:51.721292 11891 rpcz_store.cc:276] 0113 08:58:50.327331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.952518 (+625187us) service_pool.cc:225] Handling call
0113 08:58:51.720631 (+768113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:50.913192 14062 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:51.262763 14019 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:50.687219 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 585) took 1183 ms. Trace:
W20250113 08:58:51.420678 13970 consensus_meta.cc:287] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.138s user 0.009s sys 0.004s
I20250113 08:58:51.036628 14067 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:51.727272 12008 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:58:51.700470 11859 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.107s user 0.008s sys 0.034s
I20250113 08:58:50.986021 14063 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:51.040302 14051 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:50.886098 13968 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:51.039091 13958 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:51.539237 11890 rpcz_store.cc:276] 0113 08:58:48.865372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.867992 (+ 2620us) service_pool.cc:225] Handling call
0113 08:58:48.958491 (+ 90499us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:48.958669 (+ 178us) raft_consensus.cc:1529] Early marking committed up to index 2
0113 08:58:49.381517 (+422848us) log.cc:844] Serialized 14 byte log entry
0113 08:58:50.868935 (+1487418us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:50.868988 (+ 53us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:50.869034 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:50.869106 (+ 72us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:50.869248 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":792176,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":18762,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":55323,"thread_start_us":1909466,"threads_started":2,"wal-append.queue_time_us":929389,"wal-append.run_cpu_time_us":1372,"wal-append.run_wall_time_us":1037257}
I20250113 08:58:51.122189 14014 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:51.498970 14011 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.003197 14078 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:51.069366 13936 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.157946 13980 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.376490 13881 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.263509 13953 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.498840 13918 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.226804 14030 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:51.027101 14066 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:51.116777 14050 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.532984 14069 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.985819 14065 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.886230 14064 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:51.066679 14034 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.920634 14017 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:51.669333 14031 consensus_meta.cc:287] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.280s user 0.011s sys 0.003s
I20250113 08:58:50.923810 13950 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:51.491429 14029 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:50.748302 14068 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:51.788460 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.777339 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:51.827883 11934 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.003s sys 0.019s
I20250113 08:58:50.886504 14080 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:52.059094 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1792) took 1731 ms. Trace:
I20250113 08:58:50.689435 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 590) took 1216 ms. Trace:
I20250113 08:58:51.754380 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:51.122640 14036 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:52.306339 14036 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:52.306041 14014 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:52.307765 11892 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:50.953605 12040 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
W20250113 08:58:51.804203 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 55991.2us
W20250113 08:58:52.309736 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 337) took 1459 ms (client timeout 1750 ms). Trace:
W20250113 08:58:52.312068 11934 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.269s user 0.004s sys 0.024s
W20250113 08:58:51.743692 11893 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.008s sys 0.017s
I20250113 08:58:51.753906 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:51.665234 14041 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:52.312413 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.99708e+06us
W20250113 08:58:52.025449 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.010s sys 0.007s
W20250113 08:58:52.442056 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.119s user 0.011s sys 0.029s
W20250113 08:58:52.442237 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.86759e+06us
I20250113 08:58:52.443445 11859 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:52.443888 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:52.023245 11858 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.009s sys 0.019s
I20250113 08:58:52.413791 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:52.310370 11963 rpcz_store.cc:269] 0113 08:58:50.850228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.564141 (+713913us) service_pool.cc:225] Handling call
0113 08:58:52.309653 (+745512us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.445577 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2043) took 1466 ms. Trace:
I20250113 08:58:51.747907 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:52.059250 11964 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:52.446269 11968 rpcz_store.cc:276] 0113 08:58:50.979063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.723869 (+744806us) service_pool.cc:225] Handling call
0113 08:58:52.445512 (+721643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.446779 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2047) took 1466 ms. Trace:
W20250113 08:58:52.448019 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 720769us
I20250113 08:58:52.448969 12008 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:52.447971 12041 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
W20250113 08:58:51.791937 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 90690.8us
I20250113 08:58:52.444833 12685 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:51.879087 11933 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.003s sys 0.009s
I20250113 08:58:52.452165 11967 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250113 08:58:52.281045 12034 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.881s user 0.002s sys 0.012s
W20250113 08:58:52.445497 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.279s user 0.020s sys 0.022s
I20250113 08:58:52.471860 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:51.707556 13988 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.485239 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:52.460553 11893 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.649s user 0.019s sys 0.026s
I20250113 08:58:51.429771 14074 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:51.264108 14021 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.723469 13973 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:52.471742 14032 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:52.499219 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:52.475893 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:52.499651 11933 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.156s user 0.013s sys 0.018s
I20250113 08:58:51.498838 14040 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:52.460835 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2075) took 1585 ms (client timeout 1594 ms). Trace:
I20250113 08:58:52.365450 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 377) took 2882 ms. Trace:
I20250113 08:58:52.474205 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:52.389118 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:52.535502 12041 rpcz_store.cc:269] 0113 08:58:50.875849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.723030 (+847181us) service_pool.cc:225] Handling call
0113 08:58:52.460752 (+737722us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:51.939281 12006 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.004s sys 0.010s
W20250113 08:58:52.463958 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2072) took 1589 ms (client timeout 1849 ms). Trace:
W20250113 08:58:52.536438 12034 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.163s user 0.010s sys 0.014s
I20250113 08:58:52.472218 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:52.536918 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 92165.8us
I20250113 08:58:51.030207 14044 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:52.447415 11964 rpcz_store.cc:276] 0113 08:58:50.980289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.723803 (+743514us) service_pool.cc:225] Handling call
0113 08:58:52.446710 (+722907us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:51.845240 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:51.727627 13971 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:51.723116 14046 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:52.352038 12038 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s user 0.000s sys 0.020s
I20250113 08:58:52.446633 11965 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:52.501479 11968 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:52.057986 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:52.536875 12006 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.495s user 0.004s sys 0.020s
W20250113 08:58:51.873159 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.014s sys 0.009s
I20250113 08:58:52.315779 12684 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:58:52.534713 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 85780us
I20250113 08:58:52.528144 12686 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:52.491379 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:52.537544 12722 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 3
I20250113 08:58:52.536038 11889 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:58:52.536918 12042 rpcz_store.cc:269] 0113 08:58:50.874112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.380100 (+1505988us) service_pool.cc:225] Handling call
0113 08:58:52.463884 (+ 83784us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.539314 12039 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:52.538491 12685 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:51.747882 13917 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:52.473824 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:51.804909 14024 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.473331 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:52.458195 14050 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.351028 13968 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:51.799868 14084 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:52.534983 12723 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:52.534413 11891 raft_consensus.cc:2388] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:52.312274 11885 rpcz_store.cc:276] 0113 08:58:50.327923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.710039 (+1382116us) service_pool.cc:225] Handling call
0113 08:58:52.059029 (+348990us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.028740 14052 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.174876 14057 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.747030 14016 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:52.053539 14061 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:52.566385 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 123000us
I20250113 08:58:51.726964 14038 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:51.733781 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:52.567616 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.205s user 0.021s sys 0.017s
I20250113 08:58:52.223202 14034 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:52.536211 11886 rpcz_store.cc:276] 0113 08:58:49.482862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.483336 (+ 474us) service_pool.cc:225] Handling call
0113 08:58:49.812642 (+329306us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.444724 (+632082us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:58:50.686272 (+241548us) log.cc:844] Serialized 14 byte log entry
0113 08:58:51.719285 (+1033013us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:51.719337 (+ 52us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:51.719378 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:52.315496 (+596118us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54001702d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:52.315560 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:52.365352 (+ 49792us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":912070,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":307790,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":122,"spinlock_wait_cycles":1446916352,"thread_start_us":1273784,"threads_started":2,"wal-append.queue_time_us":1174855}
I20250113 08:58:52.147426 14049 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.305517 14078 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:52.173785 13964 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:52.062005 14075 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:58:52.498735 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 393) took 3595 ms (client timeout 2331 ms). Trace:
I20250113 08:58:52.567219 11888 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:58:52.568292 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 252614us
W20250113 08:58:52.934738 11888 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1796) took 2046 ms (client timeout 1822 ms). Trace:
W20250113 08:58:52.568431 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2065) took 1696 ms (client timeout 1692 ms). Trace:
W20250113 08:58:52.935320 11888 rpcz_store.cc:269] 0113 08:58:50.888165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.719248 (+831083us) service_pool.cc:225] Handling call
0113 08:58:52.934661 (+1215413us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:52.935530 12039 rpcz_store.cc:269] 0113 08:58:50.871677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.792402 (+920725us) service_pool.cc:225] Handling call
0113 08:58:52.568016 (+775614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:51.822018 14035 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:52.943714 12726 raft_consensus.cc:2388] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:52.819607 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1536) took 3856 ms. Trace:
I20250113 08:58:52.592487 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:52.937170 11889 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 389) took 2358 ms (client timeout 2490 ms). Trace:
W20250113 08:58:52.944705 12684 connection.cc:232] RPC call timeout handler was delayed by -2.2372877520000429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2331s for a response
W20250113 08:58:52.944870 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 416828us
I20250113 08:58:52.593590 11858 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:52.946187 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:58:52.937974 12038 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.437s user 0.000s sys 0.031s
W20250113 08:58:52.444646 11934 connection.cc:232] RPC call timeout handler was delayed by -2.051765006000096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20250113 08:58:52.409641 14042 consensus_meta.cc:287] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.868s user 0.004s sys 0.008s
W20250113 08:58:52.938266 11891 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1787) took 2611 ms (client timeout 3030 ms). Trace:
W20250113 08:58:52.938848 11965 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2044) took 1959 ms (client timeout 1914 ms). Trace:
I20250113 08:58:52.939009 11884 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
W20250113 08:58:52.676383 11892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1795) took 1801 ms (client timeout 2000 ms). Trace:
I20250113 08:58:52.542223 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:52.542256 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:52.650182 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:52.543064 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:52.937281 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2051) took 1678 ms (client timeout 2049 ms). Trace:
I20250113 08:58:52.682559 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 324) took 1843 ms. Trace:
W20250113 08:58:52.941424 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.90017e+06us
I20250113 08:58:52.525010 14041 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:51.734506 13977 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250113 08:58:52.419556 13923 consensus_meta.cc:287] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.929s user 0.011s sys 0.003s
W20250113 08:58:52.944715 11889 rpcz_store.cc:269] 0113 08:58:50.578844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.352480 (+1773636us) service_pool.cc:225] Handling call
0113 08:58:52.937109 (+584629us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:53.000784 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2070) took 2127 ms (client timeout 1914 ms). Trace:
I20250113 08:58:52.213642 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:52.173554 14079 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.303207 14064 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:52.089063 14039 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:51.799638 13926 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:52.499893 14031 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:52.953896 12008 connection.cc:232] RPC call timeout handler was delayed by -2.1172035540000707s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
I20250113 08:58:52.871753 14069 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:53.017794 12008 connection.cc:232] RPC call timeout handler was delayed by -2.0227035540001452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
I20250113 08:58:52.944198 14035 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:58:52.952041 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2050) took 1712 ms (client timeout 2050 ms). Trace:
W20250113 08:58:53.019138 11963 rpcz_store.cc:269] 0113 08:58:51.239277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.491013 (+1251736us) service_pool.cc:225] Handling call
0113 08:58:52.951957 (+460944us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.299230 14026 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:52.965301 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 336) took 2115 ms (client timeout 2490 ms). Trace:
I20250113 08:58:52.949535 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1785) took 2623 ms. Trace:
W20250113 08:58:52.947394 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 409073us
I20250113 08:58:52.435101 14065 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:52.851687 14036 consensus_meta.cc:287] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.545s user 0.006s sys 0.003s
W20250113 08:58:52.856393 14062 consensus_meta.cc:287] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.550s user 0.005s sys 0.006s
W20250113 08:58:52.945318 12684 connection.cc:232] RPC call timeout handler was delayed by -2.0648508140000104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
I20250113 08:58:52.943547 14017 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:52.950241 12686 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:52.499318 14025 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:53.003743 11859 connection.cc:232] RPC call timeout handler was delayed by -1.6596685749999551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
I20250113 08:58:52.944476 12034 rpcz_store.cc:276] 0113 08:58:48.910225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.910327 (+ 102us) service_pool.cc:225] Handling call
0113 08:58:49.919120 (+1008793us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:58:49.919424 (+ 304us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.988148 (+1068724us) raft_consensus.cc:1529] Early marking committed up to index 4
0113 08:58:51.373278 (+385130us) log.cc:844] Serialized 14 byte log entry
0113 08:58:52.557276 (+1183998us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:52.557349 (+ 73us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:52.557416 (+ 67us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:52.765923 (+208507us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001f1dd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:52.766009 (+ 86us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:52.766192 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19846,"prepare.queue_time_us":862804,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":19762,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":28786,"spinlock_wait_cycles":2503074176,"thread_start_us":1532809,"threads_started":2,"wal-append.queue_time_us":1054635}
W20250113 08:58:52.998692 11891 rpcz_store.cc:269] 0113 08:58:50.326389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.490107 (+2163718us) service_pool.cc:225] Handling call
0113 08:58:52.938210 (+448103us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:53.001956 11968 rpcz_store.cc:269] 0113 08:58:51.258836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.484867 (+1226031us) service_pool.cc:225] Handling call
0113 08:58:52.937205 (+452338us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:53.005345 11884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1791) took 2677 ms (client timeout 2178 ms). Trace:
I20250113 08:58:52.933300 14080 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:53.001222 12039 rpcz_store.cc:269] 0113 08:58:50.873549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.000584 (+2127035us) service_pool.cc:212] Skipping call since client already timed out
0113 08:58:53.000718 (+ 134us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250113 08:58:53.036454 11884 rpcz_store.cc:269] 0113 08:58:50.327617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.710621 (+1383004us) service_pool.cc:225] Handling call
0113 08:58:53.005264 (+1294643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.539686 14067 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:52.643523 14029 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:53.006588 11890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 393) took 2423 ms (client timeout 2984 ms). Trace:
I20250113 08:58:53.002369 12041 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
W20250113 08:58:53.007521 11934 connection.cc:232] RPC call timeout handler was delayed by -1.8199559780000527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18530000000000002s for a response
W20250113 08:58:52.594349 14015 consensus_meta.cc:287] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.523s user 0.011s sys 0.000s
I20250113 08:58:52.592365 13936 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:58:52.543893 13973 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:53.004643 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 411048us
W20250113 08:58:53.000370 11892 rpcz_store.cc:269] 0113 08:58:50.874867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.720077 (+845210us) service_pool.cc:225] Handling call
0113 08:58:52.676166 (+956089us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54000d7d54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:52.676308 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1348837760}
I20250113 08:58:53.059485 11858 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:53.003233 11965 rpcz_store.cc:269] 0113 08:58:50.979372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.723106 (+743734us) service_pool.cc:225] Handling call
0113 08:58:52.938765 (+1215659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.009007 11961 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:53.012626 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.060570 12723 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:53.059422 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.012935 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1784) took 2687 ms. Trace:
I20250113 08:58:52.542212 14101 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:52.305707 12718 rpcz_store.cc:276] 0113 08:58:49.472532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.480141 (+ 7609us) service_pool.cc:225] Handling call
0113 08:58:50.648865 (+1168724us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.595458 14056 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250113 08:58:52.875540 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 588) took 3403 ms (client timeout 1763 ms). Trace:
I20250113 08:58:52.359018 13970 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:52.444978 12721 rpcz_store.cc:276] 0113 08:58:49.467291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.468856 (+ 1565us) service_pool.cc:225] Handling call
0113 08:58:50.650144 (+1181288us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:52.944481 11856 connection.cc:232] RPC call timeout handler was delayed by -1.2647786100000076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
W20250113 08:58:53.075505 11856 connection.cc:232] RPC call timeout handler was delayed by -1.2395786099999668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
W20250113 08:58:53.075811 11856 connection.cc:232] RPC call timeout handler was delayed by -1.1513786100000516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
W20250113 08:58:53.076035 11856 connection.cc:232] RPC call timeout handler was delayed by -1.1275001920000705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
W20250113 08:58:53.076217 11856 connection.cc:232] RPC call timeout handler was delayed by -1.0298786099999688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
W20250113 08:58:53.076409 11856 connection.cc:232] RPC call timeout handler was delayed by -1.0100786100000505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20250113 08:58:52.542292 13881 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:52.543803 14032 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.016244 12006 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:52.657949 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 1989) took 3942 ms (client timeout 2064 ms). Trace:
W20250113 08:58:53.018231 12008 connection.cc:232] RPC call timeout handler was delayed by -1.9732035540000652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
I20250113 08:58:52.999437 12042 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:52.872922 13950 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:53.019995 11964 rpcz_store.cc:269] 0113 08:58:50.849938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.544517 (+1694579us) service_pool.cc:225] Handling call
0113 08:58:52.965187 (+420670us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.020466 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 390) took 2441 ms. Trace:
W20250113 08:58:53.077312 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 61144.6us
I20250113 08:58:52.594810 13988 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:53.020905 11886 rpcz_store.cc:276] 0113 08:58:50.325807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.948729 (+2622922us) service_pool.cc:225] Handling call
0113 08:58:52.949045 (+ 316us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:52.949213 (+ 168us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:52.949236 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:52.949303 (+ 67us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:52.949337 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:52.949389 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:52.949486 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:52.944288 13971 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:53.001569 13882 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:58:53.077557 11959 rpcz_store.cc:269] 0113 08:58:48.717864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.754286 (+ 36422us) service_pool.cc:225] Handling call
0113 08:58:52.657647 (+3903361us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b540039fb54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:52.657830 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4342808448}
I20250113 08:58:53.089829 13882 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.090329 13882 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:53.090478 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.091229 14063 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.092900 14067 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:52.997006 14024 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:53.004235 13918 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:53.001353 13953 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:52.735898 14014 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.001780 13977 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:52.352203 14012 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:52.949309 14052 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:53.013734 14049 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:53.024963 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1535) took 4115 ms. Trace:
W20250113 08:58:53.025714 12684 connection.cc:232] RPC call timeout handler was delayed by -1.6580508140000347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
I20250113 08:58:52.950785 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 388) took 3081 ms. Trace:
I20250113 08:58:53.026803 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.005386 13980 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:52.874120 14022 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.005857 11967 rpcz_store.cc:276] 0113 08:58:50.817915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.819949 (+ 2034us) service_pool.cc:225] Handling call
0113 08:58:50.871705 (+ 51756us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:52.461117 (+1589412us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:52.661023 (+199906us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54001ad6d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:52.661312 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3610190592}
I20250113 08:58:53.038182 12035 raft_consensus.cc:2388] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:58:53.040952 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2078) took 2164 ms (client timeout 1878 ms). Trace:
I20250113 08:58:52.852411 14066 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.944288 14016 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:52.821555 14013 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:52.948591 14075 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:53.041607 12034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2074) took 2167 ms (client timeout 2034 ms). Trace:
W20250113 08:58:53.041766 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 404) took 1968 ms (client timeout 1750 ms). Trace:
I20250113 08:58:53.042155 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 386) took 3173 ms. Trace:
I20250113 08:58:52.995539 12722 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:53.043562 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 328) took 2222 ms. Trace:
I20250113 08:58:53.045826 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.053731 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:53.019526 13926 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:53.015686 12036 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
W20250113 08:58:53.040447 11890 rpcz_store.cc:269] 0113 08:58:50.580092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.489947 (+1909855us) service_pool.cc:225] Handling call
0113 08:58:53.003218 (+513271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.023975 14035 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:52.567683 14074 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:52.944168 14068 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.059038 11960 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:53.003324 14041 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.061295 11892 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
W20250113 08:58:53.099166 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 91942.3us
I20250113 08:58:53.319603 11968 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:53.063244 11858 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:53.064558 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.064085 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:53.323380 11891 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:58:53.323802 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 260583us
W20250113 08:58:53.323359 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2042) took 2344 ms (client timeout 1566 ms). Trace:
I20250113 08:58:53.324498 11858 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:53.009356 13923 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.007966 14042 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.052284 13936 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.068087 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:53.326920 12039 rpcz_store.cc:269] 0113 08:58:51.068331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.036795 (+1968464us) service_pool.cc:212] Skipping call since client already timed out
0113 08:58:53.036890 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:58:53.327651 11965 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:53.069113 13970 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.322579 11884 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:53.066970 14034 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:53.070794 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.037994 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.000011 14084 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:53.072413 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.065014 14056 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.072968 13968 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:52.942859 12040 rpcz_store.cc:269] 0113 08:58:48.903568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.925961 (+ 22393us) service_pool.cc:225] Handling call
0113 08:58:49.889125 (+963164us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc33369360e 0000000000353db9 00007fc33b4f2a50 00007fc33b5b8814 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:52.451909 (+2562784us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23918720}
W20250113 08:58:53.080300 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 406) took 2011 ms (client timeout 1713 ms). Trace:
W20250113 08:58:53.064960 12720 rpcz_store.cc:269] 0113 08:58:49.471677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.480092 (+ 8415us) service_pool.cc:225] Handling call
0113 08:58:52.875242 (+3395150us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54007404d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:52.875415 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":250496768}
I20250113 08:58:53.082734 12006 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:53.082190 11889 rpcz_store.cc:276] 0113 08:58:50.579186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.001181 (+2421995us) service_pool.cc:225] Handling call
0113 08:58:53.019946 (+ 18765us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:53.020140 (+ 194us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:53.020167 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:53.020208 (+ 41us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:53.020244 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:53.020296 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:53.020410 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.085709 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.027678 14036 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.063771 14101 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.033718 14062 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.058558 14015 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.320519 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.085801 14061 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:53.289480 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.061383 11888 rpcz_store.cc:276] 0113 08:58:50.325429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.986323 (+2660894us) service_pool.cc:225] Handling call
0113 08:58:53.012551 (+ 26228us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:53.012707 (+ 156us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:53.012725 (+ 18us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:53.012762 (+ 37us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:58:53.012780 (+ 18us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:53.012813 (+ 33us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:53.012891 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:53.321182 12041 rpcz_store.cc:269] 0113 08:58:50.876610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.542821 (+1666211us) service_pool.cc:225] Handling call
0113 08:58:53.040689 (+497868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.399909 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.402139 11964 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:53.094154 14038 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:53.405478 14036 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:53.321743 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.405561 13882 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.406461 11892 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:53.407482 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.407837 13882 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:53.323848 12684 connection.cc:232] RPC call timeout handler was delayed by -1.5430298029999676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
W20250113 08:58:53.408532 12684 connection.cc:232] RPC call timeout handler was delayed by -1.3682508140000209s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23740000000000003s for a response
I20250113 08:58:53.408643 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.409619 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.064710 14017 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:58:53.095306 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2046) took 2115 ms (client timeout 1849 ms). Trace:
I20250113 08:58:53.410454 14017 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.410849 14017 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:53.410754 11961 rpcz_store.cc:269] 0113 08:58:50.979942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.723852 (+743910us) service_pool.cc:225] Handling call
0113 08:58:53.095105 (+1371253us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b5400197154. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.095227 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1616668032}
I20250113 08:58:53.411146 14041 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.411283 14084 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:53.298599 11934 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:53.413473 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.413813 13973 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.414265 12033 raft_consensus.cc:2363] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
W20250113 08:58:53.357992 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2076) took 2481 ms (client timeout 2068 ms). Trace:
W20250113 08:58:53.414878 12039 rpcz_store.cc:269] 0113 08:58:50.876117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.357708 (+2481591us) service_pool.cc:212] Skipping call since client already timed out
0113 08:58:53.357894 (+ 186us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250113 08:58:53.415470 12033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1533) took 4508 ms (client timeout 1511 ms). Trace:
I20250113 08:58:53.328534 11962 rpcz_store.cc:276] 0113 08:58:50.819242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.821346 (+ 2104us) service_pool.cc:225] Handling call
0113 08:58:50.857855 (+ 36509us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:53.003677 (+2145822us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:53.003710 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:53.003755 (+ 45us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:53.003790 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:53.041865 (+ 38075us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b540028e6d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:53.041916 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:53.042016 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4934484736}
I20250113 08:58:53.415822 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.416383 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.345203 12725 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:53.398469 14101 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:53.417565 13918 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.324831 11885 rpcz_store.cc:276] 0113 08:58:49.869150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.949779 (+3080629us) service_pool.cc:225] Handling call
0113 08:58:52.950159 (+ 380us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:52.950371 (+ 212us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:52.950408 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:52.950461 (+ 53us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:52.950512 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:52.950577 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:52.950714 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.399065 11959 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:53.418529 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.418655 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.282552 11790 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44358 (request call id 345) took 5193 ms. Trace:
I20250113 08:58:53.419181 12725 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
W20250113 08:58:53.318776 12034 rpcz_store.cc:269] 0113 08:58:50.875565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.041428 (+2165863us) service_pool.cc:212] Skipping call since client already timed out
0113 08:58:53.041542 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:58:53.419468 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.419927 14079 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:53.297178 11788 catalog_manager.cc:5526] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: leader changed from <none> to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:53.420478 14079 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:53.420902 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.421272 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.322175 12038 rpcz_store.cc:276] 0113 08:58:48.909075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.910015 (+ 940us) service_pool.cc:225] Handling call
0113 08:58:49.887130 (+977115us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:58:49.887375 (+ 245us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.988635 (+1101260us) raft_consensus.cc:1529] Early marking committed up to index 2
0113 08:58:51.451888 (+463253us) log.cc:844] Serialized 14 byte log entry
0113 08:58:52.997324 (+1545436us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:52.997400 (+ 76us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:52.997471 (+ 71us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:53.024743 (+ 27272us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540024efd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:53.024800 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:53.024906 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66722,"prepare.queue_time_us":882814,"prepare.run_cpu_time_us":5813,"prepare.run_wall_time_us":26754,"raft.queue_time_us":50573,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7302,"spinlock_wait_cycles":2579041280,"thread_start_us":1836811,"threads_started":2,"wal-append.queue_time_us":1177724}
I20250113 08:58:53.421608 12720 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:53.362396 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:53.324550 11960 rpcz_store.cc:269] 0113 08:58:50.978654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.197868 (+219214us) service_pool.cc:225] Handling call
0113 08:58:53.323188 (+2125320us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400196c54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.323290 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3074839296}
I20250113 08:58:53.384863 12037 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:58:53.384178 12042 rpcz_store.cc:269] 0113 08:58:51.069099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.567370 (+1498271us) service_pool.cc:225] Handling call
0113 08:58:53.080204 (+512834us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.389776 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.336400 11858 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:53.391222 12724 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 3
I20250113 08:58:53.325911 11893 rpcz_store.cc:276] 0113 08:58:49.868620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.288766 (+420146us) service_pool.cc:225] Handling call
0113 08:58:50.289143 (+ 377us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:50.721588 (+432445us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:58:52.541315 (+1819727us) log.cc:844] Serialized 14 byte log entry
0113 08:58:52.757887 (+216572us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:52.757961 (+ 74us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:52.758017 (+ 56us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:53.041946 (+283929us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540037ee54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:53.041989 (+ 43us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:53.042099 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":653641,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":125,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":100,"spinlock_wait_cycles":993923456,"thread_start_us":1946301,"threads_started":2,"wal-append.queue_time_us":482112}
W20250113 08:58:53.395254 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 312581us
I20250113 08:58:53.395714 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.391923 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.077739 14032 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:53.086834 14064 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:58:53.394249 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 444072us
I20250113 08:58:53.402549 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.344894 13971 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:53.404495 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.365350 14056 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:58:53.413596 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 115123us
I20250113 08:58:53.414227 11961 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:58:53.329660 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 308914us
I20250113 08:58:53.414433 14015 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.416482 12039 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:53.401299 13950 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:53.077919 14078 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.396927 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:53.358958 12035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 395) took 3928 ms (client timeout 2052 ms). Trace:
I20250113 08:58:53.363008 11966 raft_consensus.cc:2363] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5.
I20250113 08:58:53.359453 14034 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.129619 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.325984 13953 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.327306 13936 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:58:53.329401 14024 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.361593 14052 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:53.103741 13988 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.329367 13980 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:53.422247 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.761180 11885 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
W20250113 08:58:53.330785 12036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2071) took 2457 ms (client timeout 1827 ms). Trace:
W20250113 08:58:53.758217 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 421904us
I20250113 08:58:53.761976 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.426326 12007 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:53.762881 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 336628us
I20250113 08:58:53.759197 12041 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:58:53.421522 11788 catalog_manager.cc:5526] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 1 to 2, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 2 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
W20250113 08:58:53.765120 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1539) took 2050 ms (client timeout 2207 ms). Trace:
W20250113 08:58:53.765493 12041 rpcz_store.cc:269] 0113 08:58:51.714674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.402324 (+1687650us) service_pool.cc:225] Handling call
0113 08:58:53.765055 (+362731us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.758893 11965 raft_consensus.cc:2388] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:53.393827 12719 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:53.363822 13923 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.413417 11887 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:53.281669 14067 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:53.546717 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.758257 12686 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:53.757714 11963 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:58:53.762313 12035 rpcz_store.cc:269] 0113 08:58:49.430846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.431040 (+ 194us) service_pool.cc:225] Handling call
0113 08:58:49.890741 (+459701us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:53.358720 (+3467979us) spinlock_profiling.cc:243] Waited 2.87 s on lock 0x7b540020cbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.358887 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6649061888}
I20250113 08:58:53.789727 11884 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:53.790098 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.783098 13971 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:53.758351 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2058) took 1161 ms. Trace:
I20250113 08:58:53.694576 14016 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:53.762277 11968 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:53.420943 14079 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:53.775610 13950 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:53.551488 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.602247 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:53.828583 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2045) took 2848 ms (client timeout 1844 ms). Trace:
I20250113 08:58:53.773835 12006 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:53.085422 12717 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:53.773950 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:53.761230 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 405) took 2692 ms (client timeout 2984 ms). Trace:
W20250113 08:58:53.829922 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 56167.7us
W20250113 08:58:53.832074 12039 rpcz_store.cc:269] 0113 08:58:51.068772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.415572 (+2346800us) service_pool.cc:225] Handling call
0113 08:58:53.761176 (+345604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.803849 12721 raft_consensus.cc:2363] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
W20250113 08:58:53.415848 12033 rpcz_store.cc:269] 0113 08:58:48.906754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.906898 (+ 144us) service_pool.cc:225] Handling call
0113 08:58:49.948592 (+1041694us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3336935bc 00007fc33369317b 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:53.415241 (+3466649us) spinlock_profiling.cc:243] Waited 3.24 s on lock 0x7b54000cdad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.415378 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7465865344}
I20250113 08:58:53.777208 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.777220 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.777882 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 1998) took 1004 ms. Trace:
I20250113 08:58:53.329450 13977 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:53.776661 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2056) took 1231 ms. Trace:
I20250113 08:58:53.362650 14031 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.780723 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.774955 11962 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:53.775444 13973 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:53.728747 14012 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:53.782117 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 481) took 1308 ms. Trace:
I20250113 08:58:53.775328 13918 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:53.785540 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.789131 11889 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:53.765801 11858 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:53.790704 12035 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:53.393684 12718 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:53.790949 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.790395 12685 leader_election.cc:400] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4.
I20250113 08:58:53.798570 14067 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.766510 11788 catalog_manager.cc:5526] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129). New cstate: current_term: 3 leader_uuid: "6ea00811b7d44b49b872ac5c03b91487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:53.798239 11961 rpcz_store.cc:276] 0113 08:58:52.597038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.413188 (+816150us) service_pool.cc:225] Handling call
0113 08:58:53.758281 (+345093us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:53.798792 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 389054us
I20250113 08:58:53.758249 14042 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.728060 13926 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:53.721794 14075 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:53.761605 13953 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:53.769544 14034 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:53.384672 14013 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:53.799517 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:53.816233 12037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 401) took 4376 ms (client timeout 1600 ms). Trace:
I20250113 08:58:53.693321 14084 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:53.800405 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.801952 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:53.801191 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2053) took 1263 ms. Trace:
I20250113 08:58:53.807991 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.563302 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.673843 14062 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:53.429095 11892 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:53.779227 14068 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:58:53.826130 11887 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1788) took 3477 ms (client timeout 1763 ms). Trace:
W20250113 08:58:53.828294 12005 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33878 dropped due to backpressure: service queue is full with 50 items
I20250113 08:58:53.476480 14061 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:53.419422 11790 rpcz_store.cc:276] 0113 08:58:48.088980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.089112 (+ 132us) service_pool.cc:225] Handling call
0113 08:58:53.282453 (+5193341us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:58:49.687105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:58:49.854015 (+166910us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:58:49.854058 (+ 43us) write_op.cc:435] Acquired schema lock
0113 08:58:49.854080 (+ 22us) tablet.cc:659] Decoding operations
0113 08:58:49.854563 (+ 483us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:58:49.854619 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0113 08:58:49.854650 (+ 31us) tablet.cc:682] Acquiring locks for 2 operations
0113 08:58:49.855043 (+ 393us) tablet.cc:698] Row locks acquired
0113 08:58:49.855066 (+ 23us) write_op.cc:260] PREPARE: finished
0113 08:58:49.855229 (+ 163us) write_op.cc:270] Start()
0113 08:58:49.855354 (+ 125us) write_op.cc:276] Timestamp: P: 1736758729855184 usec, L: 0
0113 08:58:49.855379 (+ 25us) op_driver.cc:348] REPLICATION: starting
0113 08:58:50.104075 (+248696us) log.cc:844] Serialized 960 byte log entry
0113 08:58:52.054004 (+1949929us) op_driver.cc:464] REPLICATION: finished
0113 08:58:52.481215 (+427211us) write_op.cc:301] APPLY: starting
0113 08:58:52.481357 (+ 142us) tablet.cc:1364] starting BulkCheckPresence
0113 08:58:52.481519 (+ 162us) tablet.cc:1367] finished BulkCheckPresence
0113 08:58:52.481665 (+ 146us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:58:52.482101 (+ 436us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:58:52.538758 (+ 56657us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0113 08:58:52.550887 (+ 12129us) write_op.cc:312] APPLY: finished
0113 08:58:52.640801 (+ 89914us) log.cc:844] Serialized 29 byte log entry
0113 08:58:53.012118 (+371317us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:58:53.017201 (+ 5083us) write_op.cc:454] Released schema lock
0113 08:58:53.022007 (+ 4806us) write_op.cc:341] FINISH: Updating metrics
0113 08:58:53.061711 (+ 39704us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:58:53.063180 (+ 1469us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":418204,"apply.run_cpu_time_us":15819,"apply.run_wall_time_us":571186,"num_ops":2,"prepare.queue_time_us":771383,"prepare.run_cpu_time_us":47726,"prepare.run_wall_time_us":853297,"raft.queue_time_us":1377323,"raft.run_cpu_time_us":11237,"raft.run_wall_time_us":562591,"replication_time_us":2122855,"thread_start_us":2644280,"threads_started":5,"wal-append.queue_time_us":617714,"wal-append.run_cpu_time_us":46590,"wal-append.run_wall_time_us":1568993}]]}
W20250113 08:58:53.398912 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 300832us
I20250113 08:58:53.757023 14064 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.829183 11890 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
W20250113 08:58:53.829145 11966 rpcz_store.cc:269] 0113 08:58:50.979665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:51.725722 (+746057us) service_pool.cc:225] Handling call
0113 08:58:53.826404 (+2100682us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540028f0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.828509 (+ 2105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2997165824}
I20250113 08:58:53.826241 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.833087 12006 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:53.799808 14025 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:53.833186 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.776844 11934 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:53.799784 13971 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:53.835285 12038 raft_consensus.cc:2363] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
I20250113 08:58:53.835134 12041 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:53.826267 14109 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250113 08:58:53.833655 13881 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:53.836536 11965 rpcz_store.cc:276] 0113 08:58:52.773140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.328359 (+555219us) service_pool.cc:225] Handling call
0113 08:58:53.777803 (+449444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.836747 11967 rpcz_store.cc:276] 0113 08:58:52.545285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.324608 (+779323us) service_pool.cc:225] Handling call
0113 08:58:53.776599 (+451991us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:53.764099 12036 rpcz_store.cc:269] 0113 08:58:50.873811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.107686 (+1233875us) service_pool.cc:225] Handling call
0113 08:58:53.327402 (+1219716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.799216 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.739033 14101 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:53.840083 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.806154 14074 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:53.840677 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:53.840746 12007 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:53.831802 14015 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:53.841832 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:54.169620 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 403867us
I20250113 08:58:54.170297 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.170411 14084 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.171129 11967 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:54.171612 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 1996) took 1496 ms. Trace:
W20250113 08:58:54.171795 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 331102us
I20250113 08:58:54.173025 14084 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.178529 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.173641 12033 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:54.178920 14084 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:58:53.841853 13973 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:58:54.174538 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1557) took 1134 ms (client timeout 1511 ms). Trace:
I20250113 08:58:53.870343 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.180172 11962 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:53.924031 11859 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:53.928846 13988 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.181955 14107 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:54.182403 14107 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.182791 14107 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.183198 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.183394 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.183996 11961 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:54.184065 11968 rpcz_store.cc:276] 0113 08:58:52.537561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.053384 (+515823us) service_pool.cc:225] Handling call
0113 08:58:53.801132 (+747748us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.184589 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2017) took 1172 ms. Trace:
I20250113 08:58:54.184962 11961 rpcz_store.cc:276] 0113 08:58:53.011905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.182875 (+1170970us) service_pool.cc:225] Handling call
0113 08:58:54.184534 (+ 1659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.844367 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.186172 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.186729 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.186946 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1635) took 1143 ms. Trace:
I20250113 08:58:54.187608 11893 raft_consensus.cc:2388] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:54.187884 11885 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:54.188129 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1643) took 1122 ms. Trace:
I20250113 08:58:54.179821 13973 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:54.189908 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.188892 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2020) took 1150 ms. Trace:
I20250113 08:58:54.191067 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.191229 11960 rpcz_store.cc:276] 0113 08:58:53.038563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.774688 (+736125us) service_pool.cc:225] Handling call
0113 08:58:54.188821 (+414133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.192339 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.176487 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:53.830543 13950 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:54.176813 13980 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.177469 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2013) took 1296 ms. Trace:
I20250113 08:58:54.199611 14049 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.238795 11963 rpcz_store.cc:276] 0113 08:58:52.880772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.803765 (+922993us) service_pool.cc:225] Handling call
0113 08:58:54.177403 (+373638us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:53.877943 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.841653 14024 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.177163 12719 rpcz_store.cc:276] 0113 08:58:52.474114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.004009 (+529895us) service_pool.cc:225] Handling call
0113 08:58:53.782035 (+778026us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.173835 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1561) took 1091 ms. Trace:
I20250113 08:58:54.173426 14061 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.179338 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1547) took 1346 ms. Trace:
I20250113 08:58:54.257091 12033 rpcz_store.cc:276] 0113 08:58:52.833228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.840456 (+1007228us) service_pool.cc:225] Handling call
0113 08:58:54.179274 (+338818us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.258140 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.180069 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.030416 12724 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:54.206000 13988 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:54.192523 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.205811 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 482) took 1729 ms. Trace:
I20250113 08:58:54.195212 14067 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.195597 12685 leader_election.cc:403] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:58:54.285023 12718 rpcz_store.cc:276] 0113 08:58:52.474436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.070499 (+596063us) service_pool.cc:225] Handling call
0113 08:58:54.203382 (+1132883us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.197158 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1543) took 2402 ms (client timeout 1625 ms). Trace:
W20250113 08:58:54.179814 12038 rpcz_store.cc:269] 0113 08:58:53.040142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.776938 (+736796us) service_pool.cc:225] Handling call
0113 08:58:54.174469 (+397531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18304}
W20250113 08:58:54.287611 12040 rpcz_store.cc:269] 0113 08:58:51.795065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.399579 (+1604514us) service_pool.cc:225] Handling call
0113 08:58:54.197073 (+797494us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.288120 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.288658 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.288826 12038 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:54.289433 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1554) took 1268 ms. Trace:
I20250113 08:58:54.289484 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1560) took 1223 ms. Trace:
I20250113 08:58:54.289839 12038 rpcz_store.cc:276] 0113 08:58:53.021466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.287836 (+1266370us) service_pool.cc:225] Handling call
0113 08:58:54.289371 (+ 1535us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.289937 12040 rpcz_store.cc:276] 0113 08:58:53.065815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.288396 (+1222581us) service_pool.cc:225] Handling call
0113 08:58:54.289427 (+ 1031us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.198071 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.199030 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1555) took 1176 ms. Trace:
I20250113 08:58:54.173418 14015 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:54.181459 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.290968 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.203066 12007 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:54.175817 14101 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.174791 12005 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33878 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1556)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1563)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1564)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1567)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2086)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2089)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1566)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1574)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2088)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1571)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1572)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2081)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2083)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1553)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1568)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 397)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 402)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 403)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2066)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2067)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2069)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2073)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1538)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1542)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1549)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1559)
I20250113 08:58:54.288129 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.291421 12725 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:54.293462 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.293601 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.293885 12038 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:54.294195 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1551) took 1294 ms. Trace:
I20250113 08:58:54.294234 12725 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:54.294381 12038 rpcz_store.cc:276] 0113 08:58:52.999411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.293289 (+1293878us) service_pool.cc:225] Handling call
0113 08:58:54.294162 (+ 873us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.294998 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.295185 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.295686 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.295783 12725 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:54.296430 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.296624 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.199435 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:54.297544 13989 consensus_meta.cc:287] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.562s user 0.007s sys 0.009s
I20250113 08:58:54.198782 14025 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.298171 14025 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:54.175590 12037 rpcz_store.cc:269] 0113 08:58:49.439364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:49.918863 (+479499us) service_pool.cc:225] Handling call
0113 08:58:53.815933 (+3897070us) spinlock_profiling.cc:243] Waited 3.16 s on lock 0x7b54003bc754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.816137 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7278182144}
I20250113 08:58:54.298787 13989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:38492 (request call id 450) took 7465 ms. Trace:
I20250113 08:58:54.297151 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.198608 14064 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.199832 14016 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:54.299666 12038 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:54.300019 14016 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:54.300536 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.301170 12037 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
W20250113 08:58:54.259378 14017 consensus_meta.cc:287] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.848s user 0.012s sys 0.006s
W20250113 08:58:54.254667 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 421611us
I20250113 08:58:54.198935 13881 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:54.205061 14031 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:54.200641 14068 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:54.180408 12042 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:54.299794 14079 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.256906 12034 rpcz_store.cc:276] 0113 08:58:53.082225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.780461 (+698236us) service_pool.cc:225] Handling call
0113 08:58:54.173751 (+393290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.304400 14068 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.207473 14062 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:54.200505 14013 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.263278 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 469) took 1821 ms. Trace:
I20250113 08:58:54.267292 13918 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.307130 14013 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:54.299175 13989 rpcz_store.cc:276] 0113 08:58:46.832963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:46.833122 (+ 159us) service_pool.cc:225] Handling call
0113 08:58:47.063362 (+230240us) ts_tablet_manager.cc:928] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
0113 08:58:50.854365 (+3791003us) tablet_copy_client.cc:368] Tablet copy session begun
0113 08:58:53.714875 (+2860510us) tablet_metadata.cc:641] Metadata flushed
0113 08:58:53.734814 (+ 19939us) tablet_copy_client.cc:497] Wrote new tablet metadata
0113 08:58:54.297969 (+563155us) tablet_copy_client.cc:418] Wrote new consensus metadata
0113 08:58:54.298292 (+ 323us) tablet_replica.cc:206] Creating consensus instance
0113 08:58:54.298617 (+ 325us) ts_tablet_manager.cc:983] Replica registered.
0113 08:58:54.298677 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":41598,"tablet-copy.queue_time_us":131385,"thread_start_us":167636,"threads_started":2}
I20250113 08:58:54.267868 13977 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.242184 14049 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:58:54.269496 14035 consensus_meta.cc:287] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.910s user 0.012s sys 0.000s
I20250113 08:58:54.277364 14032 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.280084 13971 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.264894 14034 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.281416 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.316484 12719 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:54.317028 14049 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.317608 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.317932 14107 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.318511 12719 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:54.319450 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.319629 14049 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.320010 14107 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.320581 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.245707 12721 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
W20250113 08:58:54.284092 13936 consensus_meta.cc:287] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.507s user 0.007s sys 0.006s
I20250113 08:58:54.321597 13936 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.321919 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.322743 12722 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:54.323041 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.323661 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 591) took 1693 ms. Trace:
I20250113 08:58:54.217164 12720 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:54.324205 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.324877 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:54.282027 14056 consensus_meta.cc:287] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.520s user 0.007s sys 0.007s
I20250113 08:58:54.325140 13953 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.325584 12720 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:54.326388 12726 raft_consensus.cc:2363] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:58:54.284772 13988 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:58:53.841572 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 477) took 1368 ms (client timeout 1557 ms). Trace:
I20250113 08:58:54.286218 14112 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.286578 13973 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.292382 14015 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.332255 13881 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.333014 13973 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.328299 13988 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:54.334589 13881 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:54.292649 14024 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:54.335844 14024 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.336222 14024 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.302954 12725 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:54.302644 14016 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:54.306121 14031 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.308503 12724 rpcz_store.cc:276] 0113 08:58:52.442208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.443601 (+ 1393us) service_pool.cc:225] Handling call
0113 08:58:54.263099 (+1819498us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b54007026d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.263212 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2177076480}
I20250113 08:58:54.309317 14061 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.310142 14062 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.310952 14035 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.312943 13971 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.314178 14013 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.314694 13977 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.315740 13918 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.325901 14056 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.194161 14084 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.317507 14034 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.338953 14031 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.346697 14084 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:58:54.346992 14061 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:54.329815 13989 tablet_copy_client.cc:806] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 0 data blocks...
I20250113 08:58:54.330756 14079 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.331312 13936 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.022502 14052 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.302838 14017 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.340746 13953 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.421680 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:54.357827 13989 tablet_copy_client.cc:670] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Starting download of 1 WAL segments...
I20250113 08:58:54.324007 12722 rpcz_store.cc:276] 0113 08:58:52.630534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.320274 (+689740us) service_pool.cc:225] Handling call
0113 08:58:54.323491 (+1003217us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54004addd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.323608 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2119008128}
I20250113 08:58:54.356791 14024 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.424762 14017 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.428489 14024 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:54.199476 11887 rpcz_store.cc:269] 0113 08:58:50.326703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.437279 (+110576us) service_pool.cc:225] Handling call
0113 08:58:53.803670 (+3366391us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b5400060ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.803769 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5851320576}
I20250113 08:58:54.432258 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.234484 11890 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:54.199991 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1637) took 1142 ms. Trace:
W20250113 08:58:53.986773 11932 leader_election.cc:336] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.547s (SENT)
I20250113 08:58:54.188529 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1644) took 1105 ms. Trace:
I20250113 08:58:54.434935 11885 rpcz_store.cc:276] 0113 08:58:53.082791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.785311 (+702520us) service_pool.cc:225] Handling call
0113 08:58:54.188471 (+403160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.436612 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.390290 11886 raft_consensus.cc:2388] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:54.187369 11889 rpcz_store.cc:276] 0113 08:58:53.043145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.399375 (+356230us) service_pool.cc:225] Handling call
0113 08:58:54.186886 (+787511us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.438498 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:54.439046 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 452130us
I20250113 08:58:54.440991 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.188508 11893 rpcz_store.cc:276] 0113 08:58:53.066260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.186419 (+1120159us) service_pool.cc:225] Handling call
0113 08:58:54.188075 (+ 1656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.261188 11960 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:54.391799 11891 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:58:53.958750 11931 leader_election.cc:336] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 1.511s (SENT)
W20250113 08:58:54.178682 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 401850us
I20250113 08:58:54.186761 11961 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:54.181279 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2052) took 1644 ms. Trace:
I20250113 08:58:54.262221 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:53.868724 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.454260 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.455169 11964 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:58:54.259268 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2006) took 1380 ms. Trace:
W20250113 08:58:54.456055 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2004) took 1577 ms (client timeout 1822 ms). Trace:
I20250113 08:58:54.456190 11966 rpcz_store.cc:276] 0113 08:58:52.878879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.179793 (+1300914us) service_pool.cc:225] Handling call
0113 08:58:54.259201 (+ 79408us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.291283 11965 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:54.171906 11967 rpcz_store.cc:276] 0113 08:58:52.675271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.169976 (+1494705us) service_pool.cc:225] Handling call
0113 08:58:54.171560 (+ 1584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.298072 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.390290 11888 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:54.306052 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.302014 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.302047 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:54.402118 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 478024us
W20250113 08:58:54.291954 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 89002.5us
I20250113 08:58:54.402858 12039 raft_consensus.cc:2363] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
I20250113 08:58:54.291091 12041 rpcz_store.cc:276] 0113 08:58:53.022161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.777006 (+754845us) service_pool.cc:225] Handling call
0113 08:58:54.198962 (+421956us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.406178 12006 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:54.239943 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 162848us
I20250113 08:58:54.260264 12033 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:54.193606 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1552) took 1187 ms. Trace:
I20250113 08:58:54.264989 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:54.299151 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:54.304441 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1544) took 1625 ms (client timeout 1724 ms). Trace:
I20250113 08:58:54.404364 12005 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37184 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1566)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1574)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2088)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1571)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1572)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2081)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2083)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2094)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2100)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1553)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1568)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2090)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2091)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 397)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 402)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 403)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2066)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2067)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2069)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2073)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1538)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1542)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1549)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1559)
W20250113 08:58:54.285560 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 506975us
W20250113 08:58:54.327924 12717 rpcz_store.cc:269] 0113 08:58:52.472739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:52.473039 (+ 300us) service_pool.cc:225] Handling call
0113 08:58:53.841376 (+1368337us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400741654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:53.841491 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311482240}
I20250113 08:58:54.329435 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.316637 12723 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:58:54.328245 12726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 592) took 1691 ms (client timeout 1947 ms). Trace:
I20250113 08:58:54.337437 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 476) took 1865 ms. Trace:
I20250113 08:58:54.343118 13950 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.349725 14025 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.356112 14032 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.344600 14013 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.343565 14015 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.356745 14016 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.196691 11788 catalog_manager.cc:5526] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 reported cstate change: term changed from 2 to 3, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 3 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:58:54.199544 11858 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:54.170639 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.351734 14056 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.354223 14062 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.349571 14068 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.352953 14084 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.355243 14112 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4.
I20250113 08:58:54.353771 14031 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:54.340976 14035 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:54.442456 11889 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:54.443012 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:54.447314 11886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1639) took 1388 ms (client timeout 1557 ms). Trace:
I20250113 08:58:54.455437 11887 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:54.257472 11959 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:54.444025 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.448074 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.441192 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:54.458326 11965 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 1997) took 1746 ms (client timeout 2178 ms). Trace:
W20250113 08:58:54.444025 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2007) took 1564 ms (client timeout 2008 ms). Trace:
W20250113 08:58:54.446295 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 487464us
W20250113 08:58:54.450135 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2059) took 1818 ms (client timeout 2208 ms). Trace:
I20250113 08:58:54.445565 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1636) took 1392 ms. Trace:
I20250113 08:58:54.532416 11963 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:54.441762 11892 rpcz_store.cc:276] 0113 08:58:53.057846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.408421 (+350575us) service_pool.cc:225] Handling call
0113 08:58:54.199934 (+791513us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.450598 11962 rpcz_store.cc:276] 0113 08:58:52.537222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.840397 (+1303175us) service_pool.cc:225] Handling call
0113 08:58:54.181202 (+340805us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.579600 11965 rpcz_store.cc:269] 0113 08:58:52.712168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.197704 (+1485536us) service_pool.cc:225] Handling call
0113 08:58:54.458019 (+260315us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.580520 11960 rpcz_store.cc:269] 0113 08:58:52.879148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.192255 (+1313107us) service_pool.cc:225] Handling call
0113 08:58:54.443636 (+251381us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.456554 11964 rpcz_store.cc:269] 0113 08:58:52.878379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.834977 (+956598us) service_pool.cc:225] Handling call
0113 08:58:54.455935 (+620958us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.581307 11961 rpcz_store.cc:269] 0113 08:58:52.631299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.185912 (+1554613us) service_pool.cc:225] Handling call
0113 08:58:54.449819 (+263907us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.581904 11891 rpcz_store.cc:276] 0113 08:58:53.052866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.781049 (+728183us) service_pool.cc:225] Handling call
0113 08:58:54.445276 (+664227us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540026d254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.445468 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":456607616}
I20250113 08:58:54.458525 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:54.579437 11886 rpcz_store.cc:269] 0113 08:58:53.058407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.085371 (+ 26964us) service_pool.cc:225] Handling call
0113 08:58:54.447065 (+1361694us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b54000d8c54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.447240 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2289475840}
I20250113 08:58:54.614674 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.615077 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.615758 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:54.616333 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2061) took 1282 ms. Trace:
I20250113 08:58:54.616958 11961 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:54.616855 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:54.617573 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 212014us
W20250113 08:58:54.605484 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2014) took 1724 ms (client timeout 2064 ms). Trace:
I20250113 08:58:54.618602 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:54.619980 12721 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:54.620098 12006 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:54.620303 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 391) took 4040 ms. Trace:
I20250113 08:58:54.621479 14061 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:58:54.622099 14061 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:54.622752 14061 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:58:54.618395 11963 rpcz_store.cc:269] 0113 08:58:52.881066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.239715 (+1358649us) service_pool.cc:225] Handling call
0113 08:58:54.602445 (+362730us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.615149 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.625687 14024 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:54.627404 14024 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:54.627826 14068 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:54.627936 14024 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:58:54.628548 12042 rpcz_store.cc:269] 0113 08:58:52.679025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.760956 (+1081931us) service_pool.cc:225] Handling call
0113 08:58:54.304384 (+543428us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.630334 12718 raft_consensus.cc:2363] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:54.632193 14031 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:54.632861 14025 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:54.633337 12033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1556) took 1574 ms (client timeout 1774 ms). Trace:
I20250113 08:58:54.633736 12034 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:54.634001 14025 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:54.635279 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.634863 14025 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.635708 12005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1559) took 1577 ms. Trace:
I20250113 08:58:54.636459 12034 raft_consensus.cc:2388] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:54.617658 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2064) took 1280 ms. Trace:
I20250113 08:58:54.633860 12717 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
W20250113 08:58:54.466039 11932 leader_election.cc:336] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.634s (SENT)
I20250113 08:58:54.616816 11965 rpcz_store.cc:276] 0113 08:58:53.333633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.614492 (+1280859us) service_pool.cc:225] Handling call
0113 08:58:54.616251 (+ 1759us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.642387 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:54.633919 12033 rpcz_store.cc:269] 0113 08:58:53.027724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.257869 (+1230145us) service_pool.cc:225] Handling call
0113 08:58:54.602146 (+344277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.630355 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.641950 12722 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:54.891840 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.637478 11961 rpcz_store.cc:276] 0113 08:58:53.336892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.615333 (+1278441us) service_pool.cc:225] Handling call
0113 08:58:54.617561 (+ 2228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.746661 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.655158 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:54.749616 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2012) took 1869 ms (client timeout 2116 ms). Trace:
W20250113 08:58:54.747393 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 548513us
W20250113 08:58:54.900172 11966 rpcz_store.cc:269] 0113 08:58:52.880508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.457913 (+1577405us) service_pool.cc:225] Handling call
0113 08:58:54.749545 (+291632us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.636312 12005 rpcz_store.cc:276] 0113 08:58:53.057991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.635603 (+1577612us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:58:54.637933 12006 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:54.749650 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.620780 11891 rpcz_store.cc:276] 0113 08:58:50.579510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.618822 (+4039312us) service_pool.cc:225] Handling call
0113 08:58:54.619457 (+ 635us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:54.619911 (+ 454us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:54.619950 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:54.620006 (+ 56us) raft_consensus.cc:1640] Marking committed up to 8
0113 08:58:54.620042 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:54.620115 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:54.620224 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.653683 11893 raft_consensus.cc:2405] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:58:54.746649 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.750402 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:54.792073 11888 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1631) took 1754 ms (client timeout 1900 ms). Trace:
W20250113 08:58:54.903103 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 437079us
I20250113 08:58:54.750155 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:54.789808 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1562) took 1695 ms (client timeout 1625 ms). Trace:
I20250113 08:58:54.749941 11890 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 5
I20250113 08:58:54.647782 12035 rpcz_store.cc:276] 0113 08:58:53.006615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.789863 (+783248us) service_pool.cc:225] Handling call
0113 08:58:54.193542 (+403679us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.905668 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1654) took 1175 ms. Trace:
I20250113 08:58:54.906284 11886 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:54.646047 12036 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:58:54.651242 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2008) took 1771 ms (client timeout 2097 ms). Trace:
I20250113 08:58:54.643218 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:54.906904 11885 rpcz_store.cc:276] 0113 08:58:53.730319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.605775 (+875456us) service_pool.cc:225] Handling call
0113 08:58:54.905563 (+299788us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.648484 12040 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:54.748016 11884 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:54.906322 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1650) took 1195 ms. Trace:
I20250113 08:58:54.774050 14013 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:54.646977 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.746649 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:54.910939 11963 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:54.654948 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.912052 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.910996 11858 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:54.748317 14024 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:54.913877 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.774282 11934 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:54.914629 11887 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:54.923249 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1658) took 1191 ms. Trace:
I20250113 08:58:54.914916 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1628) took 1878 ms. Trace:
I20250113 08:58:54.915119 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 392) took 4335 ms. Trace:
I20250113 08:58:54.915346 12724 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:54.922222 11858 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:54.915483 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 267686us
I20250113 08:58:54.790982 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 598) took 1387 ms. Trace:
I20250113 08:58:54.924494 11886 rpcz_store.cc:276] 0113 08:58:50.579789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.914040 (+4334251us) service_pool.cc:225] Handling call
0113 08:58:54.914515 (+ 475us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:54.914778 (+ 263us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:54.914802 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:54.914842 (+ 40us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:54.914881 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:54.914931 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:54.915036 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.903515 11888 rpcz_store.cc:269] 0113 08:58:53.037748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.409343 (+371595us) service_pool.cc:225] Handling call
0113 08:58:54.791777 (+1382434us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b54000a3a54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.791995 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2251347840}
I20250113 08:58:54.898828 12042 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
W20250113 08:58:54.648746 12726 rpcz_store.cc:269] 0113 08:58:52.636603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.015922 (+379319us) service_pool.cc:225] Handling call
0113 08:58:54.328067 (+1312145us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54006f2a54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.328200 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2301868928}
W20250113 08:58:54.904237 12039 rpcz_store.cc:269] 0113 08:58:53.094611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.832983 (+738372us) service_pool.cc:225] Handling call
0113 08:58:54.789466 (+956483us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540002d0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.789674 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":519618944}
I20250113 08:58:54.926342 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1569) took 1267 ms. Trace:
I20250113 08:58:54.911582 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:54.917155 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.918169 11893 rpcz_store.cc:276] 0113 08:58:53.711015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.443260 (+732245us) service_pool.cc:225] Handling call
0113 08:58:54.906258 (+462998us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.921134 14032 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:54.786053 14084 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:54.909603 11968 rpcz_store.cc:269] 0113 08:58:52.879428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.297822 (+1418394us) service_pool.cc:225] Handling call
0113 08:58:54.651006 (+353184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.650702 12725 rpcz_store.cc:276] 0113 08:58:52.472437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.296408 (+1823971us) service_pool.cc:225] Handling call
0113 08:58:54.337374 (+ 40966us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.928359 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1566) took 1397 ms. Trace:
I20250113 08:58:54.654345 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.933655 12042 rpcz_store.cc:276] 0113 08:58:53.531313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.629688 (+1098375us) service_pool.cc:225] Handling call
0113 08:58:54.928269 (+298581us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.919011 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2019) took 1897 ms (client timeout 2178 ms). Trace:
W20250113 08:58:54.934309 11963 rpcz_store.cc:269] 0113 08:58:53.023767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.637021 (+1613254us) service_pool.cc:225] Handling call
0113 08:58:54.918954 (+281933us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:54.654063 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2060) took 1992 ms (client timeout 2063 ms). Trace:
I20250113 08:58:54.644462 12685 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:54.923683 11887 rpcz_store.cc:276] 0113 08:58:53.731602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.606650 (+875048us) service_pool.cc:225] Handling call
0113 08:58:54.923166 (+316516us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.649709 13950 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.652925 14016 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:54.647548 12037 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:54.936089 14062 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:54.944403 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2084) took 1151 ms. Trace:
I20250113 08:58:54.946007 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2089) took 1154 ms. Trace:
I20250113 08:58:54.946197 12033 rpcz_store.cc:276] 0113 08:58:53.790008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.913068 (+1123060us) service_pool.cc:225] Handling call
0113 08:58:54.941375 (+ 28307us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.941180 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.944628 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 344) took 1137 ms. Trace:
I20250113 08:58:54.620994 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:54.924336 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1665) took 1123 ms. Trace:
I20250113 08:58:54.949498 11892 rpcz_store.cc:276] 0113 08:58:53.800454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.642712 (+842258us) service_pool.cc:225] Handling call
0113 08:58:54.924285 (+281573us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.929003 12034 rpcz_store.cc:276] 0113 08:58:53.658767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.746236 (+1087469us) service_pool.cc:225] Handling call
0113 08:58:54.926289 (+180053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.932180 12035 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:54.929137 11889 raft_consensus.cc:2388] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:54.892896 14025 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:54.650501 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.936200 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:54.937026 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1655) took 1206 ms. Trace:
I20250113 08:58:54.924342 11885 rpcz_store.cc:276] 0113 08:58:53.036806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.913363 (+1876557us) service_pool.cc:225] Handling call
0113 08:58:54.913937 (+ 574us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:54.914357 (+ 420us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:54.914408 (+ 51us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:54.914491 (+ 83us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:54.914541 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:54.914648 (+ 107us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:54.914816 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.940172 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.946632 12037 rpcz_store.cc:276] 0113 08:58:53.791513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.301759 (+510246us) service_pool.cc:225] Handling call
0113 08:58:54.945901 (+644142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1920}
I20250113 08:58:54.944060 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.950636 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1632) took 1912 ms. Trace:
I20250113 08:58:54.922472 14013 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:54.936513 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2026) took 1150 ms. Trace:
I20250113 08:58:54.936582 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:54.944780 12038 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:54.942639 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:54.749979 14015 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:54.932614 14032 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:54.947527 11967 rpcz_store.cc:276] 0113 08:58:53.807239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.653811 (+846572us) service_pool.cc:225] Handling call
0113 08:58:54.944547 (+290736us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.646124 13988 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:54.953151 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:54.953747 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1651) took 1224 ms. Trace:
I20250113 08:58:54.944335 14024 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:54.953505 11962 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:54.997956 11960 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:54.998189 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 347) took 1167 ms. Trace:
I20250113 08:58:54.953159 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1666) took 1122 ms. Trace:
I20250113 08:58:54.999408 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.000739 12722 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:55.001152 14056 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
W20250113 08:58:54.943393 12036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1550) took 1991 ms (client timeout 1841 ms). Trace:
I20250113 08:58:55.004113 12719 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:55.004242 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1571) took 1184 ms. Trace:
I20250113 08:58:55.021399 12035 rpcz_store.cc:276] 0113 08:58:53.781729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.911151 (+1129422us) service_pool.cc:225] Handling call
0113 08:58:54.965586 (+ 54435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.004629 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:54.937644 11959 rpcz_store.cc:269] 0113 08:58:52.661989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:53.838477 (+1176488us) service_pool.cc:225] Handling call
0113 08:58:54.653848 (+815371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.013005 14032 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:54.966037 11858 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:55.009670 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.009744 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:55.010174 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 366058us
I20250113 08:58:55.027314 12723 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:58:54.993777 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 219871us
W20250113 08:58:54.965127 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 343999us
I20250113 08:58:55.012866 11893 rpcz_store.cc:276] 0113 08:58:53.729116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.944333 (+1215217us) service_pool.cc:225] Handling call
0113 08:58:54.952659 (+ 8326us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:54.952948 (+ 289us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:54.952970 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:54.953003 (+ 33us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:54.953049 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:54.953103 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:54.953209 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.927224 12718 rpcz_store.cc:276] 0113 08:58:53.403551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.287838 (+884287us) service_pool.cc:225] Handling call
0113 08:58:54.790646 (+502808us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b540077d3d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:54.790886 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":774692096}
I20250113 08:58:55.014390 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2088) took 1223 ms. Trace:
I20250113 08:58:55.014850 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1661) took 1278 ms. Trace:
I20250113 08:58:55.016798 11968 rpcz_store.cc:276] 0113 08:58:53.830373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.940843 (+1110470us) service_pool.cc:225] Handling call
0113 08:58:54.998061 (+ 57218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.000419 12008 leader_election.cc:400] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.018069 14013 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.016192 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.015444 11884 rpcz_store.cc:276] 0113 08:58:53.730622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.170389 (+439767us) service_pool.cc:225] Handling call
0113 08:58:54.931959 (+761570us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.017388 11887 rpcz_store.cc:276] 0113 08:58:53.830569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.950888 (+1120319us) service_pool.cc:225] Handling call
0113 08:58:54.952487 (+ 1599us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:54.952841 (+ 354us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:54.952876 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:54.952920 (+ 44us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:54.952952 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:54.952998 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:54.953093 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.019879 11965 rpcz_store.cc:276] 0113 08:58:53.780994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.911647 (+1130653us) service_pool.cc:225] Handling call
0113 08:58:54.931522 (+ 19875us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.015914 12037 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.032266 12038 rpcz_store.cc:276] 0113 08:58:53.791225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.654506 (+863281us) service_pool.cc:225] Handling call
0113 08:58:55.014311 (+359805us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.948540 14062 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:55.004762 12039 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
W20250113 08:58:54.943338 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1558) took 1900 ms (client timeout 1927 ms). Trace:
I20250113 08:58:55.012280 12721 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.005728 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.053627 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2081) took 1245 ms. Trace:
I20250113 08:58:55.054214 12037 rpcz_store.cc:276] 0113 08:58:53.788988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.952702 (+1163714us) service_pool.cc:225] Handling call
0113 08:58:55.034722 (+ 82020us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.022650 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.024655 14079 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
W20250113 08:58:55.025427 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 346) took 1198 ms (client timeout 1560 ms). Trace:
W20250113 08:58:55.056969 11961 rpcz_store.cc:269] 0113 08:58:53.826829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.922623 (+1095794us) service_pool.cc:225] Handling call
0113 08:58:55.025334 (+102711us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:54.964694 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:55.025787 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:55.012663 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:55.032752 12008 leader_election.cc:403] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:58:55.032351 11888 rpcz_store.cc:276] 0113 08:58:53.732590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.944727 (+1212137us) service_pool.cc:225] Handling call
0113 08:58:55.000708 (+ 55981us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.001010 (+ 302us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.001034 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.001079 (+ 45us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:58:55.001124 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.001181 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.010902 (+ 9721us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:55.027421 12005 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33878 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
W20250113 08:58:55.016319 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2018) took 2002 ms (client timeout 2128 ms). Trace:
W20250113 08:58:55.062273 11960 rpcz_store.cc:269] 0113 08:58:53.013558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.662161 (+1648603us) service_pool.cc:225] Handling call
0113 08:58:55.016265 (+354104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.011811 11889 rpcz_store.cc:276] 0113 08:58:53.038103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.613592 (+1575489us) service_pool.cc:225] Handling call
0113 08:58:54.950580 (+336988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.035305 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:55.033288 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 50086.8us
I20250113 08:58:55.040409 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1667) took 1199 ms. Trace:
I20250113 08:58:55.040899 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1572) took 1253 ms. Trace:
I20250113 08:58:55.068339 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.068653 12039 rpcz_store.cc:276] 0113 08:58:53.782056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.938057 (+1156001us) service_pool.cc:225] Handling call
0113 08:58:55.035242 (+ 97185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.043201 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52584 (request call id 1108) took 4727 ms. Trace:
I20250113 08:58:55.023684 14056 raft_consensus.cc:695] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:55.061872 12005 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33878 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2083)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2094)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2100)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1553)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1568)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2090)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2103)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2104)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2091)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2105)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 397)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 402)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 403)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2066)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2067)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2069)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2073)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2079)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1538)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1542)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1549)
I20250113 08:58:55.069968 11964 raft_consensus.cc:2388] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.069809 11891 rpcz_store.cc:276] 0113 08:58:50.329198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.012922 (+4683724us) service_pool.cc:225] Handling call
0113 08:58:55.043074 (+ 30152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.068253 11892 rpcz_store.cc:276] 0113 08:58:53.840984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.950131 (+1109147us) service_pool.cc:225] Handling call
0113 08:58:55.034746 (+ 84615us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.039999 (+ 5253us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.040044 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.040096 (+ 52us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:58:55.040134 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.040206 (+ 72us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.040339 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:55.020771 12036 rpcz_store.cc:269] 0113 08:58:52.952444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.264693 (+1312249us) service_pool.cc:225] Handling call
0113 08:58:54.937521 (+672828us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.075879 12034 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:55.076643 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2082) took 1287 ms. Trace:
I20250113 08:58:55.077033 12034 rpcz_store.cc:276] 0113 08:58:53.789344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.962365 (+1173021us) service_pool.cc:225] Handling call
0113 08:58:55.076572 (+114207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.039212 14062 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:55.038851 12040 rpcz_store.cc:269] 0113 08:58:53.041110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.290793 (+1249683us) service_pool.cc:225] Handling call
0113 08:58:54.936908 (+646115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.079370 14016 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:55.080631 14016 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:55.080708 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.081153 14016 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.079314 12726 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:55.040462 13988 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.090344 12033 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 4
W20250113 08:58:55.085847 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2011) took 2205 ms (client timeout 2118 ms). Trace:
I20250113 08:58:55.088428 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1669) took 1183 ms. Trace:
I20250113 08:58:55.083542 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1642) took 2009 ms. Trace:
I20250113 08:58:55.092171 11885 rpcz_store.cc:276] 0113 08:58:53.905408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.011685 (+1106277us) service_pool.cc:225] Handling call
0113 08:58:55.059809 (+ 48124us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.075310 (+ 15501us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.075354 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.088068 (+ 12714us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:55.088126 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.088210 (+ 84us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.088357 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.086815 11966 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
W20250113 08:58:55.078034 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 141919us
W20250113 08:58:55.093885 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 341) took 1287 ms (client timeout 1566 ms). Trace:
W20250113 08:58:55.094463 11966 rpcz_store.cc:269] 0113 08:58:53.806165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.923862 (+1117697us) service_pool.cc:225] Handling call
0113 08:58:55.093806 (+169944us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:55.095193 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 472915us
W20250113 08:58:55.095693 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 146623us
W20250113 08:58:55.096360 11964 rpcz_store.cc:269] 0113 08:58:52.880229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.614221 (+1733992us) service_pool.cc:225] Handling call
0113 08:58:55.085749 (+471528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.092916 11886 rpcz_store.cc:276] 0113 08:58:53.062409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.932510 (+1870101us) service_pool.cc:225] Handling call
0113 08:58:55.021218 (+ 88708us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.062966 (+ 41748us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.063000 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.071958 (+ 8958us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:58:55.072012 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.072097 (+ 85us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.072250 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.101672 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:55.104084 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.102963 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.104786 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.098129 11963 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:55.103967 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.106042 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 340) took 1304 ms. Trace:
I20250113 08:58:55.106397 11963 rpcz_store.cc:276] 0113 08:58:53.801147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.938275 (+1137128us) service_pool.cc:225] Handling call
0113 08:58:55.105977 (+167702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.091367 11934 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.104750 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.104979 12035 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:55.103937 11967 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:55.109026 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2023) took 1348 ms. Trace:
I20250113 08:58:55.108167 12726 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:55.109241 11967 rpcz_store.cc:276] 0113 08:58:53.760323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.011138 (+1250815us) service_pool.cc:225] Handling call
0113 08:58:55.108966 (+ 97828us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.108170 12042 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:55.110297 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.094321 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.111702 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.110584 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.112525 11961 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 1.
W20250113 08:58:55.115209 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1545) took 2353 ms (client timeout 3073 ms). Trace:
I20250113 08:58:55.116335 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 345) took 1291 ms. Trace:
W20250113 08:58:55.116643 12042 rpcz_store.cc:269] 0113 08:58:52.757147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.942791 (+2185644us) service_pool.cc:225] Handling call
0113 08:58:55.110362 (+167571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.116756 11961 rpcz_store.cc:276] 0113 08:58:53.825006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.083874 (+1258868us) service_pool.cc:225] Handling call
0113 08:58:55.116296 (+ 32422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.117951 12007 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:55.118257 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.120476 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 408) took 1328 ms. Trace:
W20250113 08:58:55.120757 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 46103.2us
I20250113 08:58:55.120926 12036 rpcz_store.cc:276] 0113 08:58:53.792173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.112666 (+1320493us) service_pool.cc:225] Handling call
0113 08:58:55.120393 (+ 7727us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.116772 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.118536 14056 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.130308 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2085) took 1340 ms. Trace:
I20250113 08:58:55.130662 12037 rpcz_store.cc:276] 0113 08:58:53.790275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.066493 (+1276218us) service_pool.cc:225] Handling call
0113 08:58:55.130244 (+ 63751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.132251 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.119244 12039 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:55.125468 12724 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
W20250113 08:58:55.119186 11932 leader_election.cc:336] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.557s (SENT)
I20250113 08:58:55.130954 11965 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.135032 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 410) took 1342 ms. Trace:
I20250113 08:58:55.136775 12034 rpcz_store.cc:276] 0113 08:58:53.792778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.096994 (+1304216us) service_pool.cc:225] Handling call
0113 08:58:55.134988 (+ 37994us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.130002 12040 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 8: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.135754 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.136687 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.139513 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.134752 11968 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:55.124694 13950 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:55.135115 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.144196 12038 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:55.142539 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.146958 12007 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:55.149868 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.116050 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.152112 12008 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.153779 13988 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:55.154564 13988 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:55.155268 11932 leader_election.cc:336] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.724s (SENT)
W20250113 08:58:55.156114 11932 leader_election.cc:336] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.868s (SENT)
I20250113 08:58:55.156538 11932 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:55.150843 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.134024 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:55.149791 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.157228 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.163008 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.125089 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.167375 11966 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.167285 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:55.168650 11839 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b300f59ff3964ee28bb6ef0c1141395e, TS 5e8883f5f83a47f4b9303dd1da4f59c0 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.168828 14084 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:55.194205 14084 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:55.194656 14084 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.194839 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2025) took 1425 ms. Trace:
I20250113 08:58:55.195297 11966 rpcz_store.cc:276] 0113 08:58:53.769434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.104449 (+1335015us) service_pool.cc:225] Handling call
0113 08:58:55.194739 (+ 90290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.196349 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.173398 11967 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.188755 12035 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.173688 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:55.189357 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b300f59ff3964ee28bb6ef0c1141395e, TS 5e8883f5f83a47f4b9303dd1da4f59c0 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.197217 11966 raft_consensus.cc:2388] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.182915 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.199270 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.199744 12039 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.200055 11966 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.201125 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.165045 12725 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.156922 11859 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.193624 11891 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 1.
I20250113 08:58:55.175662 12721 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.198186 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.198292 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.197556 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2068) took 1024 ms. Trace:
I20250113 08:58:55.201689 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:55.187585 13988 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:55.210116 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 58775.1us
W20250113 08:58:55.146567 14061 consensus_meta.cc:287] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.519s user 0.011s sys 0.000s
I20250113 08:58:55.149206 13950 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:55.162530 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2027) took 1370 ms. Trace:
I20250113 08:58:55.225446 11961 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
I20250113 08:58:55.223284 12720 raft_consensus.cc:2388] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:55.225904 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:55.226459 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.205231 12726 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.166993 12008 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.229498 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.230100 14024 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:55.229984 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.197795 11963 raft_consensus.cc:2363] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5.
I20250113 08:58:55.251277 11967 rpcz_store.cc:276] 0113 08:58:54.173340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.112813 (+939473us) service_pool.cc:225] Handling call
0113 08:58:55.197476 (+ 84663us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.190510 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.198120 11968 raft_consensus.cc:2388] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.268090 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 343) took 1461 ms. Trace:
W20250113 08:58:55.239749 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 73444.2us
W20250113 08:58:55.240487 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 86177.2us
I20250113 08:58:55.244654 12033 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:58:55.245378 11962 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:58:55.270190 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 349) took 1439 ms (client timeout 1536 ms). Trace:
W20250113 08:58:55.270679 11962 rpcz_store.cc:269] 0113 08:58:53.830975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.655587 (+824612us) service_pool.cc:225] Handling call
0113 08:58:55.270117 (+614530us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.250037 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1553) took 2233 ms. Trace:
I20250113 08:58:55.271755 12035 rpcz_store.cc:276] 0113 08:58:53.018136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.197826 (+2179690us) service_pool.cc:225] Handling call
0113 08:58:55.249973 (+ 52147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.251122 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.252117 14061 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.252332 13950 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.252774 11858 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:55.253089 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2069) took 1079 ms. Trace:
I20250113 08:58:55.275792 11961 rpcz_store.cc:276] 0113 08:58:54.173996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.117643 (+943647us) service_pool.cc:225] Handling call
0113 08:58:55.253034 (+135391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.254079 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.278942 12034 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.256651 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2065) took 1085 ms. Trace:
I20250113 08:58:55.279667 11960 rpcz_store.cc:276] 0113 08:58:54.171486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.150194 (+978708us) service_pool.cc:225] Handling call
0113 08:58:55.256591 (+106397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.258355 11964 rpcz_store.cc:276] 0113 08:58:53.792541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.105645 (+1313104us) service_pool.cc:225] Handling call
0113 08:58:55.162464 (+ 56819us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.268501 11968 rpcz_store.cc:276] 0113 08:58:53.806926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.144101 (+1337175us) service_pool.cc:225] Handling call
0113 08:58:55.268005 (+123904us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.258492 14024 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:55.258584 12722 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:58:55.249440 11890 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
W20250113 08:58:55.284246 11890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1660) took 1552 ms (client timeout 1984 ms). Trace:
W20250113 08:58:55.284723 11890 rpcz_store.cc:269] 0113 08:58:53.732213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.442348 (+710135us) service_pool.cc:225] Handling call
0113 08:58:55.284164 (+841816us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.255954 12039 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:58:55.286445 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.287231 12039 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.288338 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.203498 12037 raft_consensus.cc:2363] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.274448 14119 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:55.290086 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.166100 12718 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.276252 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.255867 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 478) took 2782 ms. Trace:
I20250113 08:58:55.280838 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2067) took 1108 ms. Trace:
I20250113 08:58:55.296505 11963 rpcz_store.cc:276] 0113 08:58:54.172753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.113202 (+940449us) service_pool.cc:225] Handling call
0113 08:58:55.280756 (+167554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.282147 12725 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:55.259063 12723 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
W20250113 08:58:55.282999 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 126943us
I20250113 08:58:55.255738 14084 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.291847 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.240000 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.293095 11966 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:55.305110 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.308058 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:55.309712 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 142831us
I20250113 08:58:55.249532 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 709) took 1057 ms. Trace:
I20250113 08:58:55.295750 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.292635 14024 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.302280 12724 raft_consensus.cc:2388] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:55.302222 12037 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.293061 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.301398 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.284605 14015 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.237179 14062 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.248721 14016 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:55.356144 11856 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 2.245s (SENT) [suppressed 80 similar messages]
I20250113 08:58:55.373714 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.356285 12719 raft_consensus.cc:2388] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:58:55.370100 11932 leader_election.cc:336] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.900s (SENT)
I20250113 08:58:55.376963 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:55.413759 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 68025.6us
I20250113 08:58:55.413945 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 609) took 1448 ms. Trace:
I20250113 08:58:55.414584 11892 raft_consensus.cc:2363] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.374560 12720 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.386852 14061 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:55.345762 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 281609us
I20250113 08:58:55.383882 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 480) took 2910 ms. Trace:
I20250113 08:58:55.414433 12719 rpcz_store.cc:276] 0113 08:58:53.965894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.412866 (+1446972us) service_pool.cc:225] Handling call
0113 08:58:55.413235 (+ 369us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.413519 (+ 284us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.413631 (+ 112us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.413686 (+ 55us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:55.413726 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.413783 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.413887 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.378378 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 594) took 2651 ms. Trace:
I20250113 08:58:55.237488 11959 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:55.374251 12036 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.373610 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.417994 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 707) took 1226 ms. Trace:
W20250113 08:58:55.418088 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 342) took 1611 ms (client timeout 2052 ms). Trace:
I20250113 08:58:55.418426 12719 rpcz_store.cc:276] 0113 08:58:54.191153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.417448 (+1226295us) service_pool.cc:225] Handling call
0113 08:58:55.417915 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.389547 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:55.388355 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:55.392294 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 600) took 1633 ms. Trace:
I20250113 08:58:55.320837 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 479) took 2829 ms. Trace:
I20250113 08:58:55.420096 12723 rpcz_store.cc:276] 0113 08:58:53.758866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.303309 (+1544443us) service_pool.cc:225] Handling call
0113 08:58:55.346443 (+ 43134us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.391836 (+ 45393us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.391873 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.391943 (+ 70us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:55.392014 (+ 71us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.392085 (+ 71us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.392227 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.320273 12042 raft_consensus.cc:2388] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.420118 11884 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:58:55.400040 14013 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.426868 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1573) took 1644 ms. Trace:
I20250113 08:58:55.416868 11961 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.369870 14120 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:55.428153 14120 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:55.428551 14120 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.388144 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.382242 14084 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.430795 12034 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:58:55.377475 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 220793us
I20250113 08:58:55.431349 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2090) took 1135 ms. Trace:
I20250113 08:58:55.418399 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2083) took 1628 ms. Trace:
I20250113 08:58:55.431847 12036 rpcz_store.cc:276] 0113 08:58:53.789750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.137173 (+1347423us) service_pool.cc:225] Handling call
0113 08:58:55.418335 (+281162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.427486 14013 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:55.348344 12008 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.433562 14013 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250113 08:58:55.433993 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 96258.7us
I20250113 08:58:55.434162 14013 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:55.439801 14013 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.439901 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.434232 14084 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:55.440646 14084 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:55.412703 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:55.445411 11962 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:58:55.399343 11839 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet fc84bb347ec2413e9188b7253b0812ab
W20250113 08:58:55.450212 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet fc84bb347ec2413e9188b7253b0812ab
I20250113 08:58:55.388440 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.357820 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.451107 12033 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.451234 11963 raft_consensus.cc:2363] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.451637 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 413) took 1158 ms. Trace:
I20250113 08:58:55.452008 12033 rpcz_store.cc:276] 0113 08:58:54.293574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.279812 (+986238us) service_pool.cc:225] Handling call
0113 08:58:55.451580 (+171768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.284796 12040 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:55.294574 14025 raft_consensus.cc:2758] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250113 08:58:55.453938 11884 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 8.
I20250113 08:58:55.292006 13988 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:55.356534 11931 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:55.456204 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 153548us
I20250113 08:58:55.456681 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.457343 11962 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.458194 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.458377 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.459187 12035 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.459729 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2092) took 1163 ms. Trace:
I20250113 08:58:55.460165 12685 leader_election.cc:400] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.460101 12035 rpcz_store.cc:276] 0113 08:58:54.296757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.279846 (+983089us) service_pool.cc:225] Handling call
0113 08:58:55.459668 (+179822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.346591 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.461903 11967 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:55.415444 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 69756.7us
I20250113 08:58:55.420856 11965 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:55.423194 12726 rpcz_store.cc:276] 0113 08:58:52.473478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.257242 (+2783764us) service_pool.cc:225] Handling call
0113 08:58:55.266693 (+ 9451us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.302843 (+ 36150us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.302871 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.302917 (+ 46us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:55.302945 (+ 28us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.303006 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.303144 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:55.414453 11932 leader_election.cc:336] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.207s (SENT)
W20250113 08:58:55.464397 11965 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2024) took 1703 ms (client timeout 2016 ms). Trace:
W20250113 08:58:55.464617 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 50210.8us
W20250113 08:58:55.464774 11965 rpcz_store.cc:269] 0113 08:58:53.760531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.062483 (+1301952us) service_pool.cc:225] Handling call
0113 08:58:55.464341 (+401858us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.465572 11962 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:58:55.320124 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 180716us
I20250113 08:58:55.373050 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.466646 14124 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:55.466831 11890 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:55.420392 11885 raft_consensus.cc:2363] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.467373 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 1995) took 4587 ms. Trace:
I20250113 08:58:55.441169 14084 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.441479 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:55.468478 11886 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:55.390710 14032 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.442384 11892 raft_consensus.cc:1235] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:55.422075 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.412887 14062 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.470439 11968 raft_consensus.cc:2388] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.471127 12007 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:55.471928 14015 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:55.472242 12726 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:55.472409 14015 raft_consensus.cc:695] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:55.469043 11932 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:55.473443 12007 leader_election.cc:400] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.473728 14124 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.416739 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 706) took 1225 ms. Trace:
I20250113 08:58:55.348497 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.348205 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.475068 12038 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:55.473466 14015 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.475695 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1568) took 1846 ms. Trace:
I20250113 08:58:55.476128 12038 rpcz_store.cc:276] 0113 08:58:53.628943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.284449 (+1655506us) service_pool.cc:225] Handling call
0113 08:58:55.475630 (+191181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.445035 11960 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.478029 11960 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:55.478610 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2041) took 4597 ms. Trace:
I20250113 08:58:55.479069 11884 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:55.478914 11960 rpcz_store.cc:276] 0113 08:58:50.881137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.477469 (+4596332us) service_pool.cc:225] Handling call
0113 08:58:55.477857 (+ 388us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.478409 (+ 552us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.478541 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.438920 14120 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.481866 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2039) took 4601 ms. Trace:
I20250113 08:58:55.482169 11964 rpcz_store.cc:276] 0113 08:58:50.880508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.474723 (+4594215us) service_pool.cc:225] Handling call
0113 08:58:55.481359 (+ 6636us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.481577 (+ 218us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.481599 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.481637 (+ 38us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:55.481665 (+ 28us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.481707 (+ 42us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.481799 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.397397 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.483584 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 1999) took 2648 ms. Trace:
I20250113 08:58:55.416119 12722 rpcz_store.cc:276] 0113 08:58:52.473816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.296405 (+2822589us) service_pool.cc:225] Handling call
0113 08:58:55.319822 (+ 23417us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.383378 (+ 63556us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.383418 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.383493 (+ 75us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:55.383579 (+ 86us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.383660 (+ 81us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.383813 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.453962 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2102) took 1051 ms. Trace:
I20250113 08:58:55.484882 12040 rpcz_store.cc:276] 0113 08:58:54.403051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.250766 (+847715us) service_pool.cc:225] Handling call
0113 08:58:55.453897 (+203131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.485900 14120 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.425171 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 722) took 1009 ms. Trace:
I20250113 08:58:55.486861 12723 rpcz_store.cc:276] 0113 08:58:54.415335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.424217 (+1008882us) service_pool.cc:225] Handling call
0113 08:58:55.424508 (+ 291us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.424830 (+ 322us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.424856 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.424896 (+ 40us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:55.424932 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.424981 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.425072 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.484059 11964 rpcz_store.cc:276] 0113 08:58:52.835347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.482745 (+2647398us) service_pool.cc:225] Handling call
0113 08:58:55.483077 (+ 332us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.483274 (+ 197us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.483303 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.483338 (+ 35us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:55.483383 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.483434 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.483529 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.416440 11934 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.446538 12034 rpcz_store.cc:276] 0113 08:58:54.296244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.292188 (+995944us) service_pool.cc:225] Handling call
0113 08:58:55.431291 (+139103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.434732 12008 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:55.448237 14013 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.278453 14079 raft_consensus.cc:695] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:55.433935 11859 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:55.449571 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 717) took 1146 ms. Trace:
I20250113 08:58:55.377487 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.417661 12725 rpcz_store.cc:276] 0113 08:58:52.727098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.321003 (+2593905us) service_pool.cc:225] Handling call
0113 08:58:55.377799 (+ 56796us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.378046 (+ 247us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.378068 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.378111 (+ 43us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:55.378140 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.378189 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.378304 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.456290 14119 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:55.460615 12685 leader_election.cc:403] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250113 08:58:55.551857 14062 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:55.345872 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.503005 11790 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44470 (request call id 245) took 1073 ms. Trace:
I20250113 08:58:55.552517 14062 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:55.415119 14016 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.445696 12724 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.553025 14062 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.553583 14013 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:55.554143 12686 leader_election.cc:336] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.743s (SENT)
I20250113 08:58:55.552780 12039 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.554729 11888 raft_consensus.cc:2363] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.552850 11790 rpcz_store.cc:276] 0113 08:58:54.427874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.428323 (+ 449us) service_pool.cc:225] Handling call
0113 08:58:55.501129 (+1072806us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4882,"reported_tablets":3}
I20250113 08:58:55.555447 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:55.555771 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 95773.7us
I20250113 08:58:55.498081 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2000) took 2621 ms. Trace:
I20250113 08:58:55.556130 12039 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.556258 11964 rpcz_store.cc:276] 0113 08:58:52.877193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.493619 (+2616426us) service_pool.cc:225] Handling call
0113 08:58:55.493992 (+ 373us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.497603 (+ 3611us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.497654 (+ 51us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.497712 (+ 58us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:58:55.497753 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.497839 (+ 86us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.497988 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.419054 12721 rpcz_store.cc:276] 0113 08:58:54.191649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.620641 (+428992us) service_pool.cc:225] Handling call
0113 08:58:55.249464 (+628823us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.557406 14109 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
W20250113 08:58:55.418578 11959 rpcz_store.cc:269] 0113 08:58:53.806586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.078737 (+1272151us) service_pool.cc:225] Handling call
0113 08:58:55.418007 (+339270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.558043 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2001) took 2680 ms. Trace:
I20250113 08:58:55.557022 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.554564 12686 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:55.529829 11788 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44354 (request call id 363) took 5225 ms. Trace:
I20250113 08:58:55.559490 12685 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:55.560324 14112 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:55.489336 12720 rpcz_store.cc:276] 0113 08:58:54.190878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.415735 (+1224857us) service_pool.cc:225] Handling call
0113 08:58:55.416107 (+ 372us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.416416 (+ 309us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.416444 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.416493 (+ 49us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:55.416531 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.416583 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.416686 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.561743 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.512396 14032 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.551891 14123 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:55.521646 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2038) took 4608 ms. Trace:
I20250113 08:58:55.562745 14119 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:55.532299 11787 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60016 (request call id 360) took 7425 ms. Trace:
W20250113 08:58:55.563562 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 129747us
I20250113 08:58:55.505101 12041 raft_consensus.cc:2388] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.551419 12719 rpcz_store.cc:276] 0113 08:58:54.303157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.423086 (+1119929us) service_pool.cc:225] Handling call
0113 08:58:55.423452 (+ 366us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.449156 (+ 25704us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.449187 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.449231 (+ 44us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:55.449286 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.449370 (+ 84us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.449495 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.562569 14123 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:55.564945 14035 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:58:55.565054 12039 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.565512 14123 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.563304 14079 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.565768 14035 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:55.562310 12034 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:55.465731 11965 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:55.296335 12717 rpcz_store.cc:276] 0113 08:58:52.473109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.254680 (+2781571us) service_pool.cc:225] Handling call
0113 08:58:55.255064 (+ 384us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.255398 (+ 334us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.255444 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.255504 (+ 60us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:58:55.255561 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.255626 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.255771 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.569661 14035 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.563174 14119 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.532748 13950 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.564687 11859 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:55.574864 13950 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:55.575441 14013 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:55.575552 13950 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250113 08:58:55.531590 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 123537us
I20250113 08:58:55.598960 13950 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:55.599327 13950 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.576016 14013 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:55.600035 14013 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.569684 11965 raft_consensus.cc:604] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:55.606112 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46296 (request call id 21) took 4726 ms. Trace:
I20250113 08:58:55.606494 11965 rpcz_store.cc:276] 0113 08:58:50.879876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.465485 (+4585609us) service_pool.cc:225] Handling call
0113 08:58:55.606012 (+140527us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.607559 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.568712 12685 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:55.611147 12724 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.457453 11931 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:58:55.565410 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 130740us
W20250113 08:58:55.612306 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 154949us
I20250113 08:58:55.612535 12008 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.612998 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:55.615332 14032 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:55.615516 14079 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:55.615828 14032 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:55.616240 14032 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.617547 11856 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:55.618348 14016 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:55.618726 14016 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:55.619076 14016 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:55.527462 14084 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.511214 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.621302 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.621686 12036 raft_consensus.cc:2363] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4.
I20250113 08:58:55.622150 13988 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:55.510555 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.625268 11856 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.625653 12033 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.626292 14056 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:58:55.626719 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.627087 14056 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:55.627564 14056 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:55.627998 14056 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.628051 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.628768 12033 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.491866 12718 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:55.629534 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 608) took 1750 ms. Trace:
I20250113 08:58:55.629964 12718 rpcz_store.cc:276] 0113 08:58:53.879095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.070386 (+1191291us) service_pool.cc:225] Handling call
0113 08:58:55.629452 (+559066us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.523018 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.578593 12007 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:55.631337 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 52852.3us
I20250113 08:58:55.631685 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.632272 12035 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.632309 12007 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:55.579027 14062 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.497892 11961 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:55.633807 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2033) took 1009 ms. Trace:
I20250113 08:58:55.634121 11961 rpcz_store.cc:276] 0113 08:58:54.624505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.276744 (+652239us) service_pool.cc:225] Handling call
0113 08:58:55.633747 (+357003us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.634763 14062 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.634939 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.635455 14015 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:55.635680 11961 raft_consensus.cc:2388] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.635959 14015 raft_consensus.cc:695] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:55.636853 14015 consensus_queue.cc:237] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.637926 14056 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.640008 14056 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.640388 14120 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:55.640956 14120 raft_consensus.cc:695] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:58:55.580017 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2087) took 1789 ms (client timeout 1762 ms). Trace:
W20250113 08:58:55.641407 12041 rpcz_store.cc:269] 0113 08:58:53.790888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:54.646438 (+855550us) service_pool.cc:225] Handling call
0113 08:58:55.579826 (+933388us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54003bc254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:55.579948 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1335314688}
I20250113 08:58:55.641957 14120 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:55.520962 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.649060 12040 raft_consensus.cc:2388] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.650166 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.650935 12040 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:55.652110 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.652357 13950 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.653673 12040 raft_consensus.cc:2388] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:58:55.653966 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 85346.2us
I20250113 08:58:55.654805 12685 leader_election.cc:400] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.655110 12685 leader_election.cc:403] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250113 08:58:55.656602 12685 leader_election.cc:400] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.656675 14121 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:55.580826 11787 rpcz_store.cc:276] 0113 08:58:48.107212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:48.107374 (+ 162us) service_pool.cc:225] Handling call
0113 08:58:55.532141 (+7424767us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41756,"reported_tablets":5}
I20250113 08:58:55.563005 11968 rpcz_store.cc:276] 0113 08:58:50.880214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.471034 (+4590820us) service_pool.cc:225] Handling call
0113 08:58:55.471362 (+ 328us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.471555 (+ 193us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.471576 (+ 21us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.471606 (+ 30us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:58:55.471630 (+ 24us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.488562 (+ 16932us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.488652 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.657290 14025 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.657626 14112 consensus_queue.cc:1046] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:55.658032 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.658087 14025 consensus_queue.cc:1046] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:58:55.658841 11968 raft_consensus.cc:2388] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.658798 14112 raft_consensus.cc:3049] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:55.659286 14025 raft_consensus.cc:3049] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:55.659404 14112 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:55.659737 14025 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:55.659934 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.660684 11968 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.660389 14112 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.427271 12042 rpcz_store.cc:276] 0113 08:58:53.782335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.122288 (+1339953us) service_pool.cc:225] Handling call
0113 08:58:55.426684 (+304396us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54003697d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:55.426811 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273082752}
I20250113 08:58:55.467736 11962 rpcz_store.cc:276] 0113 08:58:50.879561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.466444 (+4586883us) service_pool.cc:225] Handling call
0113 08:58:55.466842 (+ 398us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.467038 (+ 196us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.467060 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.467114 (+ 54us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:55.467138 (+ 24us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.467198 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.467305 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.578455 14123 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.558446 11964 rpcz_store.cc:276] 0113 08:58:52.877597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.557106 (+2679509us) service_pool.cc:225] Handling call
0113 08:58:55.557483 (+ 377us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.557706 (+ 223us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.557749 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.557806 (+ 57us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:58:55.557834 (+ 28us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.557883 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.557992 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.681939 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.683043 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.684000 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.684681 11964 raft_consensus.cc:2363] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.685523 11964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "099cf849e7414101a8741339c72fa73e"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:58:55.586046 14124 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:55.580443 14119 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.686619 11964 raft_consensus.cc:604] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:55.687897 14119 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.559597 11788 rpcz_store.cc:276] 0113 08:58:50.304089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:50.305022 (+ 933us) service_pool.cc:225] Handling call
0113 08:58:55.529629 (+5224607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:58:55.132370 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:58:55.141080 (+ 8710us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:58:55.141132 (+ 52us) write_op.cc:435] Acquired schema lock
0113 08:58:55.141161 (+ 29us) tablet.cc:659] Decoding operations
0113 08:58:55.141890 (+ 729us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:58:55.141974 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0113 08:58:55.142004 (+ 30us) tablet.cc:682] Acquiring locks for 4 operations
0113 08:58:55.183687 (+ 41683us) tablet.cc:698] Row locks acquired
0113 08:58:55.183722 (+ 35us) write_op.cc:260] PREPARE: finished
0113 08:58:55.183948 (+ 226us) write_op.cc:270] Start()
0113 08:58:55.184086 (+ 138us) write_op.cc:276] Timestamp: P: 1736758735183880 usec, L: 0
0113 08:58:55.184105 (+ 19us) op_driver.cc:348] REPLICATION: starting
0113 08:58:55.211136 (+ 27031us) log.cc:844] Serialized 1767 byte log entry
0113 08:58:55.300896 (+ 89760us) op_driver.cc:464] REPLICATION: finished
0113 08:58:55.465129 (+164233us) write_op.cc:301] APPLY: starting
0113 08:58:55.465225 (+ 96us) tablet.cc:1364] starting BulkCheckPresence
0113 08:58:55.465436 (+ 211us) tablet.cc:1367] finished BulkCheckPresence
0113 08:58:55.465571 (+ 135us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:58:55.477825 (+ 12254us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:58:55.478117 (+ 292us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0113 08:58:55.478167 (+ 50us) write_op.cc:312] APPLY: finished
0113 08:58:55.479935 (+ 1768us) log.cc:844] Serialized 41 byte log entry
0113 08:58:55.480133 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:58:55.480223 (+ 90us) write_op.cc:454] Released schema lock
0113 08:58:55.480280 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0113 08:58:55.481596 (+ 1316us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:58:55.481627 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6181,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":163844,"apply.run_cpu_time_us":5875,"apply.run_wall_time_us":16470,"num_ops":4,"prepare.queue_time_us":243703,"prepare.run_cpu_time_us":10856,"prepare.run_wall_time_us":146604,"raft.queue_time_us":18037,"raft.run_cpu_time_us":6537,"raft.run_wall_time_us":190088,"replication_time_us":99421,"thread_start_us":433260,"threads_started":4,"wal-append.queue_time_us":30868}]]}
I20250113 08:58:55.688601 12035 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.689088 11966 raft_consensus.cc:2363] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.689169 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2021) took 2617 ms. Trace:
I20250113 08:58:55.689642 11964 rpcz_store.cc:276] 0113 08:58:53.071659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.687897 (+2616238us) service_pool.cc:225] Handling call
0113 08:58:55.688512 (+ 615us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.688785 (+ 273us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.688898 (+ 113us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.688947 (+ 49us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:58:55.688977 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.689027 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.689123 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.587881 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2048) took 4607 ms. Trace:
I20250113 08:58:55.690493 14013 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.691085 11966 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:55.691838 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2063) took 2357 ms. Trace:
I20250113 08:58:55.690932 11960 rpcz_store.cc:276] 0113 08:58:50.980697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.479860 (+4499163us) service_pool.cc:225] Handling call
0113 08:58:55.480205 (+ 345us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:58:55.486240 (+ 6035us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.486275 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:58:55.486803 (+ 528us) log.cc:844] Serialized 26 byte log entry
0113 08:58:55.586162 (+ 99359us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:55.586202 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.586356 (+ 154us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:58:55.587682 (+ 1326us) raft_consensus.cc:1680] finished
0113 08:58:55.587718 (+ 36us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.587819 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5413,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":134,"thread_start_us":98891,"threads_started":1,"wal-append.queue_time_us":98464}
I20250113 08:58:55.692183 11966 rpcz_store.cc:276] 0113 08:58:53.334272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.690463 (+2356191us) service_pool.cc:225] Handling call
0113 08:58:55.690884 (+ 421us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.691610 (+ 726us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.691777 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95872}
I20250113 08:58:55.692960 11960 raft_consensus.cc:1235] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:55.584947 12034 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:58:55.695109 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.695695 12034 raft_consensus.cc:2371] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4. Re-sending same reply.
I20250113 08:58:55.695999 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250113 08:58:55.696556 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.590488 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2028) took 1382 ms. Trace:
W20250113 08:58:55.593487 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 340807us
I20250113 08:58:55.598414 11967 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:55.622225 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2091) took 1325 ms. Trace:
I20250113 08:58:55.511576 12037 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.699771 12684 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.625453 13988 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:55.696936 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 348) took 1866 ms. Trace:
I20250113 08:58:55.711324 13988 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:55.704341 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.712427 11858 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:55.712563 14123 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.712858 12035 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.713559 12036 rpcz_store.cc:276] 0113 08:58:54.296518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.510865 (+1214347us) service_pool.cc:225] Handling call
0113 08:58:55.622155 (+111290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.642542 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.508894 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.654595 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.714634 12041 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.715648 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.715917 14035 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:55.716353 14035 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:55.716450 12040 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:55.716763 14035 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.708866 13950 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.715648 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.718091 12041 raft_consensus.cc:2363] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.717831 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.714857 11963 rpcz_store.cc:276] 0113 08:58:54.207684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.451903 (+1244219us) service_pool.cc:225] Handling call
0113 08:58:55.458055 (+ 6152us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b58001c4198. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:55.590281 (+132226us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540039fb54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:55.590430 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":312658176}
I20250113 08:58:55.718922 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.719645 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.720032 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.720896 12041 raft_consensus.cc:2388] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.624248 14084 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.580932 11859 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.714658 12038 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 1.
W20250113 08:58:55.722105 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 141239us
I20250113 08:58:55.722414 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.723214 12038 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.723977 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:55.551981 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 135639us
I20250113 08:58:55.724539 14056 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:55.725029 11934 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:55.697170 12034 raft_consensus.cc:2371] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4. Re-sending same reply.
I20250113 08:58:55.725955 14123 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:55.726351 14123 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:55.726406 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.726876 14123 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.727121 12034 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:55.727205 11934 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.727989 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.728173 11934 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.661998 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.662861 14112 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.583503 14024 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:55.663332 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 48288.4us
I20250113 08:58:55.636633 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.664743 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.665776 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.721009 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.732348 12037 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.744771 11968 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:58:55.725078 14056 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:55.745775 14056 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.746371 14013 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:55.747174 12033 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.675278 11856 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.711355 11964 rpcz_store.cc:276] 0113 08:58:53.830680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.690506 (+1859826us) service_pool.cc:225] Handling call
0113 08:58:55.696460 (+ 5954us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.696645 (+ 185us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.696669 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.696705 (+ 36us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:55.696730 (+ 25us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.696777 (+ 47us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.696881 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8960}
I20250113 08:58:55.748200 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.660545 14025 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.748495 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.749325 14025 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.749428 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.750047 12033 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.702106 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.710817 11959 raft_consensus.cc:1235] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:55.656863 12685 leader_election.cc:403] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:58:55.657163 14121 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:55.758816 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2002) took 2880 ms. Trace:
I20250113 08:58:55.759686 14109 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
W20250113 08:58:55.710764 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2022) took 2382 ms (client timeout 3018 ms). Trace:
I20250113 08:58:55.680226 12007 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:55.760406 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 103952us
W20250113 08:58:55.760885 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 80751us
I20250113 08:58:55.759316 11959 rpcz_store.cc:276] 0113 08:58:52.877848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.558418 (+2680570us) service_pool.cc:225] Handling call
0113 08:58:55.558768 (+ 350us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.756328 (+197560us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.758519 (+ 2191us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54002a3d54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:55.758723 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":349181312}
I20250113 08:58:55.762845 12007 leader_election.cc:400] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.763187 12007 leader_election.cc:403] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:58:55.764000 11884 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:55.750180 14024 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:55.661867 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.765913 11892 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:55.766124 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.766341 12037 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:58:55.767244 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.767310 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.767573 14061 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.767968 12037 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:55.768059 11885 raft_consensus.cc:1235] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:55.768447 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.734043 14124 raft_consensus.cc:2758] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250113 08:58:55.768719 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.740898 14016 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.729211 11962 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.766021 14122 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:55.769645 14122 raft_consensus.cc:695] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:55.770527 14122 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.753207 12008 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.771425 14109 consensus_queue.cc:1046] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:58:55.772154 12008 leader_election.cc:400] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:58:55.769181 12042 raft_consensus.cc:2388] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.753968 11961 raft_consensus.cc:2388] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.667990 11965 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:55.733755 11858 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:55.732160 14129 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:55.729111 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.755095 12038 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.755623 11968 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:58:55.758862 14121 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:55.762894 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.764425 11960 raft_consensus.cc:2388] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:55.763618 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:55.764022 11967 rpcz_store.cc:269] 0113 08:58:53.328107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.281583 (+1953476us) service_pool.cc:225] Handling call
0113 08:58:55.710691 (+429108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.782172 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.782948 12040 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:55.753556 11963 raft_consensus.cc:2371] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4. Re-sending same reply.
I20250113 08:58:55.744201 14119 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:55.784572 14119 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:55.785022 14119 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:55.784829 14056 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:55.748324 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 73126us
I20250113 08:58:55.735601 11934 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.698624 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:58:55.714653 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.734272 13988 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:55.743175 12039 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:55.777623 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.777557 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.779273 14062 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250113 08:58:55.779389 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 45752us
I20250113 08:58:55.779726 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.777846 13950 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:58:55.806409 12033 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.749706 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:55.806326 12042 raft_consensus.cc:2388] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.806151 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:55.807101 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2105) took 1172 ms. Trace:
I20250113 08:58:55.807402 12033 rpcz_store.cc:276] 0113 08:58:54.635233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.749175 (+1113942us) service_pool.cc:225] Handling call
0113 08:58:55.807046 (+ 57871us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.806151 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:55.808706 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 73232.8us
I20250113 08:58:55.808949 12038 raft_consensus.cc:2388] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:58:55.817662 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 53888.5us
I20250113 08:58:55.820143 12007 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:55.700376 14032 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.822489 12007 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:55.820713 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 48293.5us
I20250113 08:58:55.825970 12034 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 8: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.827241 12039 raft_consensus.cc:1235] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:55.828361 12038 raft_consensus.cc:1235] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:55.829589 12037 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:55.830291 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2067) took 4956 ms. Trace:
I20250113 08:58:55.830693 12038 rpcz_store.cc:276] 0113 08:58:50.872328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.821015 (+4948687us) service_pool.cc:225] Handling call
0113 08:58:55.822752 (+ 1737us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.828756 (+ 6004us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.828942 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.785424 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 397) took 6350 ms. Trace:
I20250113 08:58:55.832700 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 402) took 6372 ms. Trace:
I20250113 08:58:55.834055 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 403) took 6373 ms. Trace:
W20250113 08:58:55.833835 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 61802.9us
I20250113 08:58:55.834319 12039 rpcz_store.cc:276] 0113 08:58:49.457390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.797722 (+6340332us) service_pool.cc:225] Handling call
0113 08:58:55.827056 (+ 29334us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.829600 (+ 2544us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.829778 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.805486 14013 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:58:55.729040 14035 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.869379 12033 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:55.870647 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2066) took 4998 ms. Trace:
W20250113 08:58:55.870882 11931 connection.cc:232] RPC call timeout handler was delayed by -1.0891730260000259s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20250113 08:58:55.819459 14137 consensus_queue.cc:1046] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250113 08:58:55.833086 12040 rpcz_store.cc:276] 0113 08:58:49.434813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.783567 (+6348754us) service_pool.cc:225] Handling call
0113 08:58:55.783952 (+ 385us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.784950 (+ 998us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.784996 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.785052 (+ 56us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:55.785093 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.785150 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.785279 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.836575 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:55.836872 14139 consensus_queue.cc:1046] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250113 08:58:55.836340 12008 leader_election.cc:400] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:55.838335 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 398) took 6402 ms. Trace:
I20250113 08:58:55.843155 11856 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:55.806195 14062 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:55.845433 11934 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:55.847774 14121 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.848618 11858 leader_election.cc:400] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5.
I20250113 08:58:55.877586 11858 leader_election.cc:403] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:58:55.825695 14124 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:55.829046 14015 raft_consensus.cc:2758] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20250113 08:58:55.861420 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.828574 12036 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.870941 14035 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:55.840075 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 103470us
I20250113 08:58:55.872395 12038 raft_consensus.cc:1235] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:55.871397 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.874055 12041 rpcz_store.cc:276] 0113 08:58:49.435978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.809541 (+6373563us) service_pool.cc:225] Handling call
0113 08:58:55.837812 (+ 28271us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.838003 (+ 191us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.838026 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.838071 (+ 45us) raft_consensus.cc:1640] Marking committed up to 8
0113 08:58:55.838101 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.838148 (+ 47us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.838252 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.871019 12033 rpcz_store.cc:276] 0113 08:58:50.872083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.821525 (+4949442us) service_pool.cc:225] Handling call
0113 08:58:55.830509 (+ 8984us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.869866 (+ 39357us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.870424 (+ 558us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b54001fdbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:55.870585 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55766400}
W20250113 08:58:55.869390 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 93314.8us
I20250113 08:58:55.872601 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:55.884729 12684 leader_election.cc:400] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:55.885730 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1538) took 4171 ms. Trace:
I20250113 08:58:55.885959 12684 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.849514 12686 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:55.860734 12035 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:55.872260 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:55.887513 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2073) took 5013 ms. Trace:
I20250113 08:58:55.887802 11892 raft_consensus.cc:2388] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:55.834447 12042 rpcz_store.cc:276] 0113 08:58:49.461090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.821733 (+6360643us) service_pool.cc:225] Handling call
0113 08:58:55.833573 (+ 11840us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.833761 (+ 188us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.833791 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.833830 (+ 39us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:55.833856 (+ 26us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.833915 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.833996 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:55.873905 11931 connection.cc:232] RPC call timeout handler was delayed by -1.088677682000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
I20250113 08:58:55.871896 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:55.888635 11931 leader_election.cc:336] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 1.547s (SENT)
I20250113 08:58:55.849975 12034 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:55.877398 14062 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:55.831944 14112 raft_consensus.cc:3049] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:55.882972 12039 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250113 08:58:55.884680 11884 raft_consensus.cc:2371] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5. Re-sending same reply.
I20250113 08:58:55.887881 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:55.889039 11931 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:55.875250 14120 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
W20250113 08:58:55.897883 11859 consensus_peers.cc:487] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:55.898555 14142 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250113 08:58:55.898815 11931 leader_election.cc:336] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 1.625s (SENT)
I20250113 08:58:55.899448 11931 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.886161 12040 rpcz_store.cc:276] 0113 08:58:51.714300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.884703 (+4170403us) service_pool.cc:225] Handling call
0113 08:58:55.885124 (+ 421us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.885364 (+ 240us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.885394 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.885434 (+ 40us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:58:55.885468 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.885524 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.885630 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.876255 12007 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:55.902094 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 69100.9us
I20250113 08:58:55.903334 14122 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:55.887917 12038 rpcz_store.cc:276] 0113 08:58:50.874361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.854772 (+4980411us) service_pool.cc:225] Handling call
0113 08:58:55.872199 (+ 17427us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.887271 (+ 15072us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.887461 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.868657 14013 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:55.893602 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2079) took 5016 ms. Trace:
I20250113 08:58:55.893610 12035 raft_consensus.cc:1235] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
W20250113 08:58:55.894393 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 59328.9us
I20250113 08:58:55.891767 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:55.886396 14121 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:55.849143 14123 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.891698 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2069) took 5018 ms. Trace:
I20250113 08:58:55.881551 14129 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:55.904754 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:55.905694 11932 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:55.879518 14109 consensus_queue.cc:1046] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20250113 08:58:55.886874 14079 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250113 08:58:55.953711 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 71124.3us
I20250113 08:58:55.953398 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1541) took 4238 ms. Trace:
I20250113 08:58:55.964632 12038 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:55.983506 12039 rpcz_store.cc:276] 0113 08:58:50.876923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.860957 (+4984034us) service_pool.cc:225] Handling call
0113 08:58:55.880326 (+ 19369us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:58:55.893269 (+ 12943us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.893534 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.983892 12033 rpcz_store.cc:276] 0113 08:58:51.715306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.890229 (+4174923us) service_pool.cc:225] Handling call
0113 08:58:55.900095 (+ 9866us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.948607 (+ 48512us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.948644 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.952994 (+ 4350us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:58:55.953053 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.953144 (+ 91us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.953316 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:55.954102 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 55364.2us
W20250113 08:58:55.959195 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 127528us
I20250113 08:58:55.979705 12037 raft_consensus.cc:1235] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:55.986441 12041 raft_consensus.cc:2388] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:55.985424 12034 rpcz_store.cc:276] 0113 08:58:50.872904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.826581 (+4953677us) service_pool.cc:225] Handling call
0113 08:58:55.849804 (+ 23223us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.891002 (+ 41198us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.891637 (+ 635us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:55.991832 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 160417us
I20250113 08:58:55.893630 14112 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:55.993091 11856 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:55.997174 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1542) took 4261 ms. Trace:
I20250113 08:58:55.993868 11888 raft_consensus.cc:2388] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:55.995735 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1540) took 4271 ms. Trace:
I20250113 08:58:55.997750 12035 rpcz_store.cc:276] 0113 08:58:51.746999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.892663 (+4145664us) service_pool.cc:225] Handling call
0113 08:58:55.893439 (+ 776us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.986663 (+ 93224us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.986898 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.997988 12042 rpcz_store.cc:276] 0113 08:58:51.714983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.890050 (+4175067us) service_pool.cc:225] Handling call
0113 08:58:55.960503 (+ 70453us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:55.985893 (+ 25390us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:55.985922 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:55.985967 (+ 45us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:58:55.986006 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:55.986068 (+ 62us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:55.986173 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:55.908643 14120 raft_consensus.cc:695] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:55.992802 14119 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:55.999269 12008 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:55.999836 14120 consensus_queue.cc:237] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:56.001412 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 175337us
I20250113 08:58:56.001762 14137 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:56.001863 12008 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:58:56.005277 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 114831us
W20250113 08:58:56.006525 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 120625us
W20250113 08:58:56.006829 11931 leader_election.cc:336] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 1.724s (SENT)
I20250113 08:58:56.007391 11931 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:55.954960 14035 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:55.959192 14142 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:55.986994 14013 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:58:55.985756 13988 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:55.987275 14079 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:55.983383 14129 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:56.015866 11934 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.017653 11964 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:56.001157 14119 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:56.002323 14137 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:58:56.031023 12686 leader_election.cc:336] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.052s (SENT)
I20250113 08:58:56.010596 14122 raft_consensus.cc:695] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:58:56.020802 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 122166us
I20250113 08:58:56.011412 14032 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:56.013622 14016 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.018548 14015 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:56.019003 14079 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.018256 14056 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.016659 14124 raft_consensus.cc:695] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:56.021277 14035 raft_consensus.cc:695] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:56.031944 14137 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.039744 14123 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:56.058163 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 51016.4us
I20250113 08:58:56.058841 12684 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:58:56.059372 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 47136.4us
I20250113 08:58:56.019346 14129 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.036183 13988 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5.
I20250113 08:58:56.045663 14142 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:58:56.065860 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 72891.6us
I20250113 08:58:56.067930 11856 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:56.062744 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1549) took 3111 ms. Trace:
I20250113 08:58:56.071336 14138 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:56.071858 14138 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:56.071756 14035 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.073230 14141 consensus_queue.cc:1046] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20250113 08:58:56.073854 14024 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:56.074723 14141 raft_consensus.cc:3049] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:56.075122 14141 raft_consensus.cc:738] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:56.069348 14025 raft_consensus.cc:3049] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:56.075929 14025 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:56.072386 14138 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:56.081602 11859 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:56.056535 14016 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:56.094102 14016 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:56.094569 14016 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.076161 14141 consensus_queue.cc:260] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.095541 14141 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:56.096530 11960 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:56.071328 12036 rpcz_store.cc:276] 0113 08:58:52.943875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:55.906857 (+2962982us) service_pool.cc:225] Handling call
0113 08:58:56.013680 (+106823us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:56.038141 (+ 24461us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:56.039658 (+ 1517us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:56.047559 (+ 7901us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:58:56.047623 (+ 64us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:56.047703 (+ 80us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:56.055442 (+ 7739us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:56.076886 14025 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.089404 14149 raft_consensus.cc:2743] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:56.097708 14143 consensus_queue.cc:1046] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250113 08:58:56.098407 14025 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.098685 14032 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:58:56.091882 11885 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:56.104655 11934 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:56.111673 14056 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:56.112181 14056 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:56.112586 14056 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.062642 14061 raft_consensus.cc:3049] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:56.114909 14061 raft_consensus.cc:738] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:56.115763 14061 consensus_queue.cc:260] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:56.106815 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 92497.3us
I20250113 08:58:56.117365 14109 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:56.069324 14123 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:56.119127 14138 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.120086 14148 raft_consensus.cc:3049] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:56.120472 14148 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:56.121631 14138 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.121315 14148 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.067039 14142 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:56.122664 14142 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:56.122727 14148 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:58:56.122068 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 54199.9us
W20250113 08:58:56.125881 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 119114us
I20250113 08:58:56.127048 11856 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:56.137982 14124 consensus_queue.cc:237] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:56.148507 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 112091us
W20250113 08:58:56.151914 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 50519.9us
I20250113 08:58:56.166545 11856 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:56.148452 14061 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:56.172463 14150 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:56.150460 14141 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:56.150216 13988 raft_consensus.cc:3049] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:56.166249 14138 consensus_queue.cc:1046] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
W20250113 08:58:56.179591 11931 leader_election.cc:336] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 1.927s (SENT)
I20250113 08:58:56.184262 11931 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:58:56.185544 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 129823us
W20250113 08:58:56.181494 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 122706us
I20250113 08:58:56.187333 12684 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:56.178580 13988 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:56.200399 14152 raft_consensus.cc:988] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: : Instructing follower 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:56.200631 11968 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:56.200912 14152 raft_consensus.cc:1076] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Signalling peer 6ea00811b7d44b49b872ac5c03b91487 to start an election
I20250113 08:58:56.202854 11885 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20250113 08:58:56.203938 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 65194us
I20250113 08:58:56.195991 13988 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:58:56.205262 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 49694us
I20250113 08:58:56.207690 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.208340 12718 raft_consensus.cc:2363] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:58:56.208495 14144 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:56.209084 14144 raft_consensus.cc:695] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:56.210013 14144 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.211791 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.173276 14123 raft_consensus.cc:2758] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250113 08:58:56.176689 14150 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:56.219360 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.226502 12686 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:56.255824 14112 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.258303 14079 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.262502 12008 leader_election.cc:400] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:58:56.170861 14137 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.260511 14079 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:56.217633 14149 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250113 08:58:56.277355 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 275194us
I20250113 08:58:56.245777 12725 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:56.148870 14122 consensus_queue.cc:237] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:56.272748 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 67178.8us
I20250113 08:58:56.262934 12008 leader_election.cc:403] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250113 08:58:56.223847 14150 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.261401 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:56.278925 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 52555.1us
W20250113 08:58:56.290577 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 111044us
I20250113 08:58:56.277768 14035 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:58:56.222882 14084 consensus_queue.cc:1046] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20250113 08:58:56.234122 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.227316 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.302526 14016 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.338785 14084 raft_consensus.cc:3049] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:56.339232 14084 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:56.340121 14084 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:56.331914 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 158938us
I20250113 08:58:56.255033 14121 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20250113 08:58:56.340777 14016 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:56.278563 14141 raft_consensus.cc:695] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:58:56.333487 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 265329us
I20250113 08:58:56.337141 12686 leader_election.cc:400] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3.
I20250113 08:58:56.340629 11884 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:56.341426 13989 tablet_copy_client.cc:538] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:58:56.277863 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.343340 11859 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:56.344352 11963 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:56.347164 14062 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:56.348975 11968 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250113 08:58:56.331485 13950 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:58:56.351362 11932 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:56.354876 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45224.2us
W20250113 08:58:56.326472 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 64102.2us
I20250113 08:58:56.366257 12684 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.277978 14142 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:56.356305 11931 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.303481 14056 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:56.357717 14084 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:56.358273 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.361562 14035 consensus_queue.cc:1046] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:56.369545 14112 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.370267 12008 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:56.370611 13988 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.371171 14062 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.366956 14120 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20250113 08:58:56.374363 14015 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250113 08:58:56.377699 14157 log.cc:926] Time spent T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.075s user 0.002s sys 0.001s
I20250113 08:58:56.396485 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.308931 14013 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.401654 14163 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:56.402123 14163 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:56.402501 14163 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.402145 11892 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:58:56.402977 14013 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:56.405436 14167 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:56.407086 14056 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:56.407125 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 240704us
I20250113 08:58:56.383320 14119 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:56.366320 11934 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:56.380532 13950 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250113 08:58:56.381667 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 45386us
I20250113 08:58:56.383783 14144 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:58:56.381680 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 56067.8us
I20250113 08:58:56.412324 11932 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.413080 14121 consensus_queue.cc:1046] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250113 08:58:56.413602 14139 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:56.414280 14121 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:58:56.414161 14139 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.415741 14142 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.416311 14139 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.416402 14142 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:56.385488 12008 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:56.417506 14119 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:56.418041 14119 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:56.419904 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:56.421154 14120 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:58:56.421175 12719 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:56.385977 14166 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:56.422473 14166 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:56.422878 14166 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.386180 14156 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:56.387161 14035 raft_consensus.cc:3049] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:56.388520 14129 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:56.384502 11931 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.382723 12718 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.395689 14137 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:56.414657 14121 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:58:56.424247 12008 consensus_peers.cc:487] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 -> Peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Couldn't send request to peer 5f9c33777f6845c2848a2a6102a1c2ff. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:58:56.382568 12684 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.428915 14167 raft_consensus.cc:695] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:56.428882 14035 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:56.389477 14141 consensus_queue.cc:237] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.439554 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:58:56.678385 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 322245us
I20250113 08:58:56.678957 11964 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.679201 11934 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.661737 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.451689 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.654341 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.662680 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.680492 11965 raft_consensus.cc:2388] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:56.663090 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:56.681146 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 256996us
I20250113 08:58:56.622895 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.681804 12008 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:56.631047 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.441288 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.662249 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:56.549355 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 102285us
I20250113 08:58:56.510033 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:58:56.688953 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 278271us
I20250113 08:58:56.689997 11856 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:56.509543 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.690949 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:56.691488 11968 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.693360 11887 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:56.693928 11886 raft_consensus.cc:2363] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:56.696208 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.705197 11932 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.696940 11962 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.707082 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:58:56.708240 14168 log.cc:926] Time spent T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20250113 08:58:56.708428 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.708850 11892 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:56.709520 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.707094 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.707172 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:56.710637 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 328136us
I20250113 08:58:56.711298 11891 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:56.709563 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.598974 14163 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:56.713425 14138 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:56.714035 14163 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.714229 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.716650 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.718490 14129 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:56.704995 14150 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.719309 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.714851 11889 raft_consensus.cc:2388] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:56.721280 14150 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:56.722080 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.722879 14035 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.724184 14035 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.724262 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.712046 12684 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.728477 14112 consensus_queue.cc:1046] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:58:56.722105 11885 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:56.731428 11964 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:56.731722 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.732626 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:58:56.731431 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 86931.6us
I20250113 08:58:56.742194 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.744614 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.745283 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.746085 11890 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:56.745869 11964 raft_consensus.cc:2388] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:58:56.750099 11886 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:56.699133 14084 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:56.753887 11961 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:56.697976 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:56.761890 11968 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:56.769623 11893 raft_consensus.cc:1235] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
W20250113 08:58:56.769776 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 79877.8us
I20250113 08:58:56.695554 14119 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:56.753554 14139 raft_consensus.cc:3049] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:56.749045 14150 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:58:56.780299 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 181441us
I20250113 08:58:56.781162 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.751232 13989 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap starting.
I20250113 08:58:56.752581 12718 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:56.780794 11962 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:56.770360 11959 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:56.748778 14032 consensus_queue.cc:1046] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20250113 08:58:56.772086 12719 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:56.798117 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.783650 14139 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:56.829442 12008 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:56.793467 14025 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:56.830354 11893 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
W20250113 08:58:56.833539 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 104400us
I20250113 08:58:56.838773 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:56.844612 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 63086us
I20250113 08:58:56.845314 11856 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:56.848320 11886 raft_consensus.cc:2388] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:56.839666 14061 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:56.853147 11856 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:56.855461 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:56.856128 14061 raft_consensus.cc:695] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:56.789419 14167 consensus_queue.cc:237] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.854140 14032 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250113 08:58:56.858944 14129 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:56.861757 11886 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
W20250113 08:58:56.862632 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 478173us
I20250113 08:58:56.871945 11931 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.865603 14152 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:58:56.865451 14139 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:56.866492 14169 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:56.867017 14180 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:58:56.870000 14175 log.cc:926] Time spent T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
W20250113 08:58:56.854892 14174 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20250113 08:58:56.919350 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 66320.2us
I20250113 08:58:56.853849 12720 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:56.920534 11960 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:56.920960 11966 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:56.921905 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:56.924783 12006 leader_election.cc:336] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.763s (SENT)
W20250113 08:58:56.926180 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 54313us
I20250113 08:58:56.863262 14162 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:58:56.928758 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 223650us
I20250113 08:58:56.931521 11931 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:56.925683 14123 raft_consensus.cc:2758] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250113 08:58:56.920703 14129 raft_consensus.cc:695] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:58:56.931161 14144 consensus_meta.cc:287] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.505s user 0.009s sys 0.000s
I20250113 08:58:56.923727 14013 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:58:56.973603 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:56.935199 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 256060us
I20250113 08:58:56.961660 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:56.953495 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:56.949440 12686 leader_election.cc:336] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.523s (SENT)
I20250113 08:58:56.953234 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:56.985018 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 53582us
I20250113 08:58:56.985929 11966 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:58:56.928754 14035 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:56.987717 11891 raft_consensus.cc:2388] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:56.961576 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:56.934888 11856 leader_election.cc:400] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4.
I20250113 08:58:57.014189 11856 leader_election.cc:403] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250113 08:58:56.953620 14160 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:56.953908 14166 consensus_meta.cc:287] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.531s user 0.003s sys 0.008s
I20250113 08:58:56.943053 11888 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "099cf849e7414101a8741339c72fa73e"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:52600
I20250113 08:58:57.017467 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:57.023335 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 73950.4us
I20250113 08:58:57.016855 11889 raft_consensus.cc:2388] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:58:57.026700 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 92769.4us
I20250113 08:58:56.985448 14144 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:57.023640 11888 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:58:57.058211 11888 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:56.927721 14180 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.011878 14061 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.014993 13950 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:57.032073 14161 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:57.033313 11962 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:57.033365 14183 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:57.032446 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:56.985065 14119 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:57.034240 11959 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:58:57.048424 14153 consensus_queue.cc:1046] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:58:57.021068 14035 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.015309 14013 raft_consensus.cc:695] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:57.051826 14160 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:56.988962 14112 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:58:57.056016 12686 leader_election.cc:336] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.536s (SENT)
W20250113 08:58:57.109048 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 244449us
I20250113 08:58:57.021580 14123 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250113 08:58:57.111052 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 55052.2us
W20250113 08:58:57.118419 12686 leader_election.cc:336] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.560s (SENT)
I20250113 08:58:57.118873 12686 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:57.112627 11931 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:57.059507 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
W20250113 08:58:57.046746 14185 log.cc:926] Time spent T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250113 08:58:57.024956 14032 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20250113 08:58:57.125542 14013 consensus_queue.cc:237] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:57.047252 14165 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:57.068503 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:56.973753 14162 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.076921 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:57.078148 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 93503.8us
I20250113 08:58:57.067680 12719 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:57.066910 14152 raft_consensus.cc:695] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:57.074072 14169 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:57.065032 13950 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:57.137933 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 80582.1us
I20250113 08:58:57.059067 14166 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:57.122151 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.122808 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.140067 11890 raft_consensus.cc:2363] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5.
I20250113 08:58:57.066295 14129 consensus_queue.cc:237] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.140728 14015 raft_consensus.cc:3049] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:58:57.119812 14112 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.128341 11889 raft_consensus.cc:2405] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250113 08:58:57.130910 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:57.169435 12725 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:57.170164 11856 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:57.171142 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:57.132807 12686 leader_election.cc:336] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.566s (SENT)
I20250113 08:58:57.135286 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.165894 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.137437 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.109795 14120 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250113 08:58:57.074286 14183 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.174551 11885 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 6
I20250113 08:58:57.175408 11891 raft_consensus.cc:2363] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
W20250113 08:58:57.237593 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 129095us
I20250113 08:58:57.173805 12686 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:58:57.613206 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 304870us
I20250113 08:58:57.348968 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:58:57.177744 11856 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:58:57.175935 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:57.256829 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 144272us
I20250113 08:58:57.301545 12725 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:57.253546 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:57.361315 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:57.619990 12726 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:58:57.255295 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 78331.4us
I20250113 08:58:57.133898 14056 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:57.459239 11888 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.141330 14015 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:57.133443 14194 consensus_queue.cc:1046] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.592s
I20250113 08:58:57.293699 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:57.255053 14152 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.280315 14169 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:58.193557 14169 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.171691 14196 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:57.654132 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.635717 11888 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:58.197582 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.06474e+06us
W20250113 08:58:57.975580 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 347709us
I20250113 08:58:58.079990 12726 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:57.670074 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.996505 11885 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:58:57.856554 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:58.194032 14205 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:57.856608 14150 consensus_meta.cc:287] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.062s user 0.006s sys 0.004s
I20250113 08:58:58.229120 14150 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:58.230024 14150 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.230523 14150 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.232527 14150 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:57.753643 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:58.234187 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.234905 11887 raft_consensus.cc:2388] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.235682 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.236672 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.236856 11884 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:58.095521 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:57.627076 11931 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:58:58.238467 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 611447us
I20250113 08:58:58.239578 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:52600 (request call id 1680) took 1287 ms. Trace:
I20250113 08:58:57.472617 14035 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:57.627302 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 354761us
I20250113 08:58:58.203943 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:57.671416 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:58.203962 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:57.714594 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 536870us
I20250113 08:58:57.165942 14123 raft_consensus.cc:2758] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250113 08:58:57.847443 12723 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:58:57.654592 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:57.678921 12722 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:58:57.252184 14186 log.cc:926] Time spent T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20250113 08:58:57.254693 14199 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20250113 08:58:57.920609 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 126112us
I20250113 08:58:58.019379 14025 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.220942 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:57.905978 14161 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.014s sys 0.016s
I20250113 08:58:58.221881 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1827) took 1087 ms. Trace:
I20250113 08:58:58.146378 14124 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:58:58.110540 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 485678us
W20250113 08:58:58.247058 14156 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.013s sys 0.002s
I20250113 08:58:57.448621 13950 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:57.474390 14165 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4.
I20250113 08:58:58.240057 11888 rpcz_store.cc:276] 0113 08:58:56.932081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:56.942869 (+ 10788us) service_pool.cc:225] Handling call
0113 08:58:58.219127 (+1276258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:58.318563 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.251840 14187 consensus_queue.cc:1046] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 1.031s
I20250113 08:58:58.250200 11856 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:57.478266 14162 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.265910 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:57.626120 14167 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:58.267984 11892 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20250113 08:58:58.347783 11892 pending_rounds.cc:77] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 3
I20250113 08:58:58.258908 11887 raft_consensus.cc:2363] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:58.274224 14150 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:57.382601 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:58.275444 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 735) took 1081 ms. Trace:
I20250113 08:58:58.375319 12722 rpcz_store.cc:276] 0113 08:58:57.168273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.175548 (+ 7275us) service_pool.cc:225] Handling call
0113 08:58:58.249983 (+1074435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:58.359755 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:58.276012 12041 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:58.276580 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 734) took 1115 ms. Trace:
I20250113 08:58:58.377027 12723 rpcz_store.cc:276] 0113 08:58:57.134029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.134187 (+ 158us) service_pool.cc:225] Handling call
0113 08:58:58.249602 (+1115415us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:58.221307 12724 raft_consensus.cc:2388] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:58.095027 14201 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:58:58.145772 14181 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:57.557654 14180 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:58.144695 14032 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:57.646648 14166 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:57.669309 14141 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:58:58.284804 11885 rpcz_store.cc:276] 0113 08:58:57.134593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.134788 (+ 195us) service_pool.cc:225] Handling call
0113 08:58:58.221792 (+1087004us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:58.204181 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:58.288676 11886 raft_consensus.cc:2363] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3.
I20250113 08:58:57.715610 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:57.754565 14183 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:57.890641 14203 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:58:58.413731 14203 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:57.626828 14056 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:58:58.416615 14056 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:58.416347 14032 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.417560 12721 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:58.418175 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 739) took 1058 ms. Trace:
I20250113 08:58:58.418790 14032 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.315768 14124 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.418520 12721 rpcz_store.cc:276] 0113 08:58:57.360006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.589554 (+229548us) service_pool.cc:225] Handling call
0113 08:58:58.418108 (+828554us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:58:58.336875 14013 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.003s sys 0.019s
I20250113 08:58:57.765100 11891 raft_consensus.cc:1235] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 2. Current term is 3. Ops: []
W20250113 08:58:58.421172 14013 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s user 0.003s sys 0.020s
I20250113 08:58:58.422248 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:58.422515 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.423233 11891 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:58.399689 14181 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.423511 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.424286 11886 raft_consensus.cc:2388] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:58:58.290478 12684 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.013s sys 0.052s
W20250113 08:58:58.424686 12684 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.803s user 0.014s sys 0.052s
I20250113 08:58:57.461552 14112 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:58.425071 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.279258 14188 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.385440 14167 raft_consensus.cc:695] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:58.425825 14188 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:58.424433 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 157060us
W20250113 08:58:58.316406 12687 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.014s sys 0.041s
I20250113 08:58:58.349397 12726 raft_consensus.cc:1235] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:58.365813 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.366225 14209 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.194989 14015 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.340785 13950 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.370689 14208 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.372206 14162 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250113 08:58:58.373162 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 123042us
W20250113 08:58:58.296950 12720 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.015s sys 0.021s
I20250113 08:58:58.416365 14203 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:58.379801 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 736) took 1175 ms. Trace:
I20250113 08:58:58.373488 11892 pending_rounds.cc:99] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20250113 08:58:58.468341 11892 raft_consensus.cc:2883] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:58:58.399791 12037 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:58.401595 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.404676 14141 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:58:58.469575 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 246938us
I20250113 08:58:58.424121 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:57.670037 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:58:57.254503 14122 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.993s
I20250113 08:58:58.425853 11886 raft_consensus.cc:2388] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.427038 14167 consensus_queue.cc:237] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:58.288120 14161 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.157s user 0.016s sys 0.027s
I20250113 08:58:58.431320 11859 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:58.321686 14206 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:58.347499 14123 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:58.382871 14150 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.478857 11887 raft_consensus.cc:2388] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.595470 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:58.595827 11893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1818) took 1663 ms (client timeout 1671 ms). Trace:
I20250113 08:58:58.596146 11887 raft_consensus.cc:2371] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3. Re-sending same reply.
W20250113 08:58:58.596284 11893 rpcz_store.cc:269] 0113 08:58:56.932355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:56.932508 (+ 153us) service_pool.cc:225] Handling call
0113 08:58:58.595645 (+1663137us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b540038ed54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:58.595769 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3306167680}
I20250113 08:58:58.597278 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.454344 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:58.247635 14205 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.601071 14205 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.291415 14025 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.463644 11889 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:58:58.316766 14156 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.013s sys 0.002s
I20250113 08:58:58.456714 12684 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:58.602921 14025 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:58:58.603355 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 146714us
I20250113 08:58:58.603482 14025 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:58.603915 14025 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:58.604236 14025 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:58.605551 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:58.470306 12686 connection.cc:232] RPC call timeout handler was delayed by -1.0704497890001221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
I20250113 08:58:58.467646 12724 rpcz_store.cc:276] 0113 08:58:57.204615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.204841 (+ 226us) service_pool.cc:225] Handling call
0113 08:58:57.856325 (+651484us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:58.379731 (+523406us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82191616}
I20250113 08:58:58.606001 11788 catalog_manager.cc:5526] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 4, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:58.427911 14219 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.618435 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:58.619122 12724 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:58.618822 14219 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.619347 11962 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:58.618227 14205 raft_consensus.cc:2758] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20250113 08:58:58.425237 14181 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:58.526569 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.597870 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.546425 14084 raft_consensus.cc:3049] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:58.551255 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:58.454368 14152 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:58.539242 12036 raft_consensus.cc:2388] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:58:58.405690 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:58:58.660714 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 230961us
I20250113 08:58:58.591564 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:58:58.613870 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 136562us
I20250113 08:58:58.661515 11884 raft_consensus.cc:2371] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4. Re-sending same reply.
I20250113 08:58:58.607465 14121 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.545982 14200 raft_consensus.cc:3049] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Stepping down as leader of term 3
W20250113 08:58:58.467387 12720 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s user 0.016s sys 0.021s
I20250113 08:58:58.617414 12038 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250113 08:58:58.608062 14203 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.593771 14056 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:58.617868 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 186633us
I20250113 08:58:58.701798 11963 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:58.532712 14015 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:58.608193 11788 catalog_manager.cc:5526] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:58.622541 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:58.710633 11968 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:58.623157 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.711987 11890 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:58.550479 12035 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:58.713347 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 445) took 1035 ms. Trace:
I20250113 08:58:58.713785 12035 rpcz_store.cc:276] 0113 08:58:57.678044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.678177 (+ 133us) service_pool.cc:225] Handling call
0113 08:58:58.549723 (+871546us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:58:58.549977 (+ 254us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:58.713004 (+163027us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:58.713273 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6076288}
I20250113 08:58:58.627079 11887 raft_consensus.cc:2388] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.644533 11892 raft_consensus.cc:2388] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.718199 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:58:58.631336 11889 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1828) took 1392 ms (client timeout 1516 ms). Trace:
W20250113 08:58:58.648379 12006 leader_election.cc:336] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.508s (SENT)
I20250113 08:58:58.535202 14137 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.186s
I20250113 08:58:58.644841 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:58.605616 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:58.662072 12686 leader_election.cc:336] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.563s (SENT)
I20250113 08:58:58.613759 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:58.662801 12041 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:58.746050 12686 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:58.740854 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:58.746783 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.672735 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.661671 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:58.702946 11856 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:58.552265 11959 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
W20250113 08:58:58.428416 12687 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.810s user 0.014s sys 0.041s
I20250113 08:58:58.703290 11859 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:58.701193 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.705070 11961 raft_consensus.cc:1235] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:58.705557 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 444) took 1028 ms. Trace:
I20250113 08:58:58.705098 11962 raft_consensus.cc:1235] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:58.465312 14162 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:58.621438 14112 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.477577 14129 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.476331 14141 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:58.621032 14161 raft_consensus.cc:695] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:58.587342 12719 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:58.465601 14188 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.730675 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:58:58.768659 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:58:58.769461 11885 raft_consensus.cc:2363] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5.
I20250113 08:58:58.769433 11966 raft_consensus.cc:3049] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:58.708760 11788 catalog_manager.cc:5526] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 2 to 4, leader changed from 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:58.720681 11931 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:58.735817 12007 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:58.737871 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.012s sys 0.020s
I20250113 08:58:58.644479 14084 raft_consensus.cc:738] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:58:58.651549 14153 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.749984 14013 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:58.645821 12034 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:58:58.738914 12006 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:58.753134 14221 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.741585 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.745229 14124 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.657438 14139 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:58.743336 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:58.463871 14209 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.454661 14169 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.748760 11886 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.701565 14200 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:58.752002 11892 raft_consensus.cc:2388] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.752678 12038 rpcz_store.cc:276] 0113 08:58:57.677509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.677665 (+ 156us) service_pool.cc:225] Handling call
0113 08:58:58.554755 (+877090us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:58:58.617034 (+ 62279us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:58.705290 (+ 88256us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:58.705494 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8112768}
W20250113 08:58:58.753700 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 91648.7us
I20250113 08:58:58.754346 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:58.466387 13950 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:58.761662 11965 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:58.756390 11893 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:58:58.643728 14123 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.736982 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 440) took 1083 ms. Trace:
I20250113 08:58:58.764088 11884 raft_consensus.cc:1235] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 3. Ops: []
I20250113 08:58:58.757704 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:58:58.758914 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 95033.1us
I20250113 08:58:58.794950 12722 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:58.770053 11966 raft_consensus.cc:738] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:58.770393 11887 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 1. Current term is 2. Ops: []
I20250113 08:58:58.778604 14205 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
W20250113 08:58:58.738744 11889 rpcz_store.cc:269] 0113 08:58:57.239268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.626320 (+387052us) service_pool.cc:225] Handling call
0113 08:58:58.631080 (+1004760us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b5400060ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:58.631258 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":353690496}
I20250113 08:58:58.819886 11856 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:58:58.772017 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.013s sys 0.021s
I20250113 08:58:58.753983 12721 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:58.763921 14035 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20250113 08:58:58.773859 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:58.846398 12723 raft_consensus.cc:2388] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:58:58.774374 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 442) took 1136 ms. Trace:
I20250113 08:58:58.775326 11885 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:58:58.775806 14219 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.774603 11788 catalog_manager.cc:5526] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:58.776294 14084 consensus_queue.cc:260] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:58.777194 14120 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.719151 14203 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.742039 12726 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:58:58.778069 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:58.467083 14208 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.759966 14180 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.781795 11890 raft_consensus.cc:1235] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:58:58.763273 14165 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.760499 14162 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:58.664970 14152 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:58.784744 12686 leader_election.cc:336] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.761s (SENT)
I20250113 08:58:58.921677 12686 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:58.785430 14181 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250113 08:58:58.786026 14217 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:58:58.786502 14213 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:58.786991 14212 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.787016 11886 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:58.783102 14121 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:58:58.803944 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68200.5us
W20250113 08:58:58.806080 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 157747us
I20250113 08:58:58.806232 14182 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.805791 14161 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.806668 14032 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:58.914566 14183 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:58.807185 14061 raft_consensus.cc:2743] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:58:58.801676 12036 rpcz_store.cc:276] 0113 08:58:57.637361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.669707 (+ 32346us) service_pool.cc:225] Handling call
0113 08:58:58.705824 (+1036117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:58.807657 12717 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:58.808243 14123 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.809553 11892 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:58.749559 14187 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:58.760625 14201 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
W20250113 08:58:58.774317 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 737) took 1496 ms (client timeout 1669 ms). Trace:
I20250113 08:58:58.778611 14166 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:58.962152 14230 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
W20250113 08:58:58.962240 14231 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20250113 08:58:58.780753 14188 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:58:58.763773 14112 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.961227 14216 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:58.849159 11788 catalog_manager.cc:5526] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 2 to 4, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:58:58.947211 14032 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:58:58.854799 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 151599us
I20250113 08:58:58.969913 11859 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:58.809947 13950 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:58.858114 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.906978 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.905356 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:58.769917 14129 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:58:58.913681 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:58.923192 12722 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:58.974089 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:58.974879 12722 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:58:58.975901 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:58:58.977108 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:58.923628 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:58:58.978606 11934 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:58.848907 14205 raft_consensus.cc:695] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:58.783787 14013 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.972522 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:58.984542 11959 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:58.985492 14013 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.986116 14208 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.982349 11893 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:58.986671 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 639) took 1858 ms. Trace:
I20250113 08:58:58.987044 12718 rpcz_store.cc:276] 0113 08:58:57.128305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.130896 (+ 2591us) service_pool.cc:225] Handling call
0113 08:58:58.182664 (+1051768us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc33369360e 0000000000353db9 00007fc33b4f2a50 00007fc33b5b8814 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:58:58.182942 (+ 278us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:58:58.183451 (+ 509us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:58.183506 (+ 55us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:58:58.184143 (+ 637us) log.cc:844] Serialized 26 byte log entry
0113 08:58:58.647076 (+462933us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:58.647139 (+ 63us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:58.647243 (+ 104us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:58:58.976400 (+329157us) raft_consensus.cc:1680] finished
0113 08:58:58.976427 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:58.986607 (+ 10180us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4192,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":147,"spinlock_wait_cycles":49624832,"thread_start_us":454488,"threads_started":1,"wal-append.queue_time_us":554175}
I20250113 08:58:58.989939 14205 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:58.936197 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 215619us
I20250113 08:58:58.937351 12039 raft_consensus.cc:2388] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:58:58.938367 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 143155us
W20250113 08:58:58.940348 12006 leader_election.cc:336] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.518s (SENT)
I20250113 08:58:58.941227 12007 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:59.009920 11931 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:58:59.010246 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 69137.3us
W20250113 08:58:59.010316 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 70006us
I20250113 08:58:58.806811 14220 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:59.014685 14244 consensus_queue.cc:1046] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:58:58.946669 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:59.016505 12724 raft_consensus.cc:2388] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.933840 14084 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:58.948582 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:59.017953 11965 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:58.924469 14056 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:58.925055 14150 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:58.957583 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 172862us
I20250113 08:58:58.844626 11966 consensus_queue.cc:260] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.656857 14196 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:58:58.961156 14232 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:58:58.959997 12719 rpcz_store.cc:269] 0113 08:58:57.277978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.280118 (+ 2140us) service_pool.cc:225] Handling call
0113 08:58:58.772478 (+1492360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:58:59.027570 14015 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250113 08:58:58.864096 14120 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.037377 14232 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.038241 14244 raft_consensus.cc:3049] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:59.038695 14244 raft_consensus.cc:738] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:59.039855 14232 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:58.850672 12034 rpcz_store.cc:276] 0113 08:58:57.638356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.654854 (+ 16498us) service_pool.cc:225] Handling call
0113 08:58:58.774315 (+1119461us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3840}
I20250113 08:58:59.039635 14244 consensus_queue.cc:260] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:58.945415 14182 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.041188 14244 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:58.949266 14212 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.042511 14182 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:59.044075 14212 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:58.963364 14187 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.046553 12726 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:58:58.914871 14180 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:58.948453 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 636) took 1934 ms. Trace:
I20250113 08:58:58.971910 12723 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:58:58.949553 14213 raft_consensus.cc:695] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:58.979815 12722 raft_consensus.cc:2388] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:58:58.981029 11934 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:58.972149 14032 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:59.013316 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:59.014123 11856 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:58.961779 14166 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:59.023152 12686 leader_election.cc:336] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.855s (SENT)
I20250113 08:58:58.986121 14013 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:58.803197 14221 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.027776 12007 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:58:59.104024 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 76324.5us
I20250113 08:58:59.104755 12007 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:59.030226 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:58.912685 14165 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.017747 13950 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:59.033491 14218 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:59.033833 14239 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:58:58.777885 14153 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.036662 11931 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:59.113934 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:59.037788 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:58.968345 11788 catalog_manager.cc:5526] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:58:59.014863 14220 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:59.062518 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 81537.8us
I20250113 08:58:58.922262 14200 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:59.084647 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 70605.8us
I20250113 08:58:59.064900 11966 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 3
I20250113 08:58:59.133525 11934 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:59.101495 14188 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:59.079895 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:59.100868 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 58879.2us
I20250113 08:58:59.139183 12723 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:58:59.090955 14216 raft_consensus.cc:695] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:59.047222 14112 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:59.036656 14152 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:59.110448 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:59.113889 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 135541us
I20250113 08:58:59.147120 11856 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:59.084201 14123 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:59.099924 14056 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:58:59.125748 12722 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:58:59.107059 14061 consensus_queue.cc:1046] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
W20250113 08:58:59.090440 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 67309.3us
I20250113 08:58:59.093919 12684 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:59.136175 12042 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:59.115991 14122 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:59.137148 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:59.137859 12724 raft_consensus.cc:1235] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
W20250113 08:58:59.141112 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 104537us
I20250113 08:58:59.141376 11934 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:59.154840 11965 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:59.141661 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:59.143196 12006 leader_election.cc:400] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:59.156481 12006 leader_election.cc:403] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:58:59.110635 14013 raft_consensus.cc:695] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:59.145500 12685 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:59.114204 14203 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:59.146499 12034 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:58:59.159940 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:59.161900 12724 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:58:59.100229 14209 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:59.156924 12007 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:58:59.158953 13950 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:58:59.153601 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 59813.4us
I20250113 08:58:59.142385 14217 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:58:59.169013 14252 consensus_queue.cc:1046] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:58:59.163782 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:59.169945 13950 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:59.170243 12724 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:59.164822 12685 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:59.137652 12719 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:58:59.167456 14203 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:59.150535 14123 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:59.168645 14246 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:58:59.159886 14112 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:58:59.154805 14150 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:59.178273 14198 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4.
I20250113 08:58:59.179636 12684 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:59.153743 14239 raft_consensus.cc:695] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:58:59.183812 11966 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:58:59.160434 14056 raft_consensus.cc:695] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:59.160506 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:58:59.171221 14165 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:59.193444 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 52149.8us
I20250113 08:58:59.193984 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:58:59.194784 12724 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:58:59.121850 14218 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.191686 12723 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:58:59.184047 12685 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:58:59.190902 14112 raft_consensus.cc:695] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:58:59.061142 12720 rpcz_store.cc:276] 0113 08:58:57.009485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.012210 (+ 2725us) service_pool.cc:225] Handling call
0113 08:58:57.027868 (+ 15658us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:58:57.175018 (+147150us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:57.177096 (+ 2078us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:58:57.670029 (+492933us) log.cc:844] Serialized 26 byte log entry
0113 08:58:58.721111 (+1051082us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:58.855961 (+134850us) log.cc:844] Serialized 14 byte log entry
0113 08:58:58.913095 (+ 57134us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:58.917411 (+ 4316us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:58:58.924316 (+ 6905us) raft_consensus.cc:1680] finished
0113 08:58:58.933529 (+ 9213us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:58.943355 (+ 9826us) inbound_call.cc:173] Queueing success response
0113 08:58:59.040532 (+ 97177us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b18003865c0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d1204 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
Metrics: {"prepare.queue_time_us":101111,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":3217,"raft.queue_time_us":130045,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":13066,"spinlock_wait_cycles":16094464,"thread_start_us":1222376,"threads_started":3,"wal-append.queue_time_us":899936}
I20250113 08:58:59.179469 14123 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:59.199726 14056 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:59.209333 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 68789.1us
I20250113 08:58:59.188148 14150 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:59.187522 14209 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:58:59.204811 14252 raft_consensus.cc:3049] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Stepping down as leader of term 1
W20250113 08:58:59.211494 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 731) took 2081 ms (client timeout 1548 ms). Trace:
I20250113 08:58:59.212584 14242 consensus_queue.cc:1046] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20250113 08:58:59.213336 14235 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:59.189960 14232 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:58:59.209367 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 49881us
I20250113 08:58:59.215435 12006 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:58:59.213902 11934 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:58:59.227274 12723 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250113 08:58:59.222379 12717 rpcz_store.cc:269] 0113 08:58:57.130113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.133684 (+ 3571us) service_pool.cc:225] Handling call
0113 08:58:59.211290 (+2077606us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54006004d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:59.211411 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":544399616}
I20250113 08:58:59.229982 12006 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:58:59.207366 14217 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
W20250113 08:58:59.236793 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 484906us
I20250113 08:58:59.237607 11858 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:58:59.238250 14124 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:58:59.238746 14124 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:59.239208 14124 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:58:59.238989 14190 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:58:59.241518 14208 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:58:59.241685 14245 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:59.241959 14220 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:59.242326 14245 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:59.242794 14245 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:59.243387 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:58:59.216869 14150 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:59.217378 14209 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:59.213860 14013 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.219137 14252 raft_consensus.cc:738] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:59.208034 14165 consensus_queue.cc:1046] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20250113 08:58:59.187703 14216 consensus_queue.cc:237] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:58:59.251413 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 88435us
I20250113 08:58:59.219807 14137 consensus_queue.cc:1046] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20250113 08:58:59.164041 14213 consensus_queue.cc:237] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:58:59.222723 14152 consensus_queue.cc:1046] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
W20250113 08:58:59.252100 12686 leader_election.cc:336] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.291s (SENT)
I20250113 08:58:59.801754 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:58:59.802613 13989 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.561s user 0.011s sys 0.013s
I20250113 08:58:59.356957 14242 raft_consensus.cc:3049] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:59.806048 14242 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:58:59.807437 14242 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:59.800246 14124 consensus_meta.cc:287] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.543s user 0.010s sys 0.002s
I20250113 08:58:59.392670 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:58:59.809530 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 579582us
W20250113 08:58:59.809680 14032 consensus_meta.cc:287] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.686s user 0.000s sys 0.012s
W20250113 08:58:59.809983 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 441) took 2172 ms (client timeout 2121 ms). Trace:
W20250113 08:58:59.366940 14025 consensus_meta.cc:287] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.762s user 0.006s sys 0.007s
I20250113 08:58:59.338676 13950 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:58:59.252799 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 448) took 1508 ms (client timeout 1594 ms). Trace:
I20250113 08:58:59.328656 14239 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.234601 14232 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:58:59.811406 14139 consensus_meta.cc:287] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.027s user 0.007s sys 0.008s
I20250113 08:58:59.258828 14166 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:58:59.811990 14245 consensus_meta.cc:287] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.569s user 0.004s sys 0.006s
I20250113 08:58:59.812105 14209 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:58:59.234772 14235 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:58:59.812300 14255 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.265s user 0.001s sys 0.000s
I20250113 08:58:59.354050 14122 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:58:59.257321 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:58:59.814308 14203 consensus_meta.cc:287] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.632s user 0.011s sys 0.000s
W20250113 08:58:59.815083 14141 consensus_meta.cc:287] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.045s user 0.013s sys 0.000s
I20250113 08:58:59.337811 14190 consensus_queue.cc:1046] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.477s
I20250113 08:58:59.391788 14219 raft_consensus.cc:3049] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:58:59.235136 14180 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:59.338676 14112 consensus_queue.cc:237] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.820746 12008 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
W20250113 08:58:59.865027 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 612904us
W20250113 08:58:59.826484 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 588937us
W20250113 08:58:59.877851 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.003s sys 0.021s
W20250113 08:59:00.023126 12039 rpcz_store.cc:269] 0113 08:58:57.744710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.744837 (+ 127us) service_pool.cc:225] Handling call
0113 08:58:58.551074 (+806237us) spinlock_profiling.cc:243] Waited 5.82 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:58:59.249174 (+698100us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54003688d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:59.252719 (+ 3545us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":686898432}
W20250113 08:58:59.670655 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 453553us
I20250113 08:59:00.026448 12006 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:00.028875 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 208233us
W20250113 08:59:00.026705 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.841s user 0.003s sys 0.032s
W20250113 08:59:00.031834 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 850285us
W20250113 08:59:00.025913 12042 rpcz_store.cc:269] 0113 08:58:57.637883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.654273 (+ 16390us) service_pool.cc:225] Handling call
0113 08:58:59.248832 (+1594559us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x7b540020d5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:58:59.809879 (+561047us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1655427840}
I20250113 08:59:00.037458 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:00.031073 11858 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:58:59.808975 14162 consensus_meta.cc:287] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.871s user 0.006s sys 0.006s
W20250113 08:58:59.781634 14123 consensus_meta.cc:287] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.571s user 0.009s sys 0.002s
W20250113 08:58:59.561210 12721 consensus_meta.cc:287] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.704s user 0.010s sys 0.003s
I20250113 08:58:59.236079 14187 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:58:59.817687 14200 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:58:59.618450 11859 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.012s sys 0.024s
W20250113 08:59:00.053272 11859 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.081s user 0.018s sys 0.035s
I20250113 08:58:59.390771 14153 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:58:59.637688 14244 consensus_meta.cc:287] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.590s user 0.013s sys 0.000s
I20250113 08:59:00.080415 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:58:59.781987 14121 consensus_meta.cc:287] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.836s user 0.012s sys 0.000s
I20250113 08:58:59.439599 14129 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:00.033737 14232 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:00.033908 14235 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:00.037871 14203 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:58:59.461652 14167 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:00.032029 12008 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
W20250113 08:58:59.901115 14205 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.007s sys 0.025s
W20250113 08:59:00.051789 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.014s sys 0.020s
W20250113 08:59:00.186676 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.925s user 0.014s sys 0.020s
I20250113 08:58:59.289450 14217 raft_consensus.cc:695] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:00.053696 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.08382e+06us
I20250113 08:58:59.582923 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2076) took 1090 ms. Trace:
I20250113 08:58:59.345348 14169 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:00.192250 11967 rpcz_store.cc:276] 0113 08:58:58.467966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:58.468231 (+ 265us) service_pool.cc:225] Handling call
0113 08:58:58.529194 (+ 60963us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:58:58.943566 (+414372us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:58:59.091651 (+148085us) log.cc:844] Serialized 14 byte log entry
0113 08:58:59.183315 (+ 91664us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:58:59.183384 (+ 69us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:58:59.183432 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:59.558443 (+375011us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b540006e0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:58:59.558520 (+ 77us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:58:59.558713 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12397,"prepare.queue_time_us":31273,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":42202,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":6825,"spinlock_wait_cycles":952411520,"thread_start_us":194984,"threads_started":2,"wal-append.queue_time_us":290014}
I20250113 08:58:59.437386 14221 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:58:59.569988 14212 raft_consensus.cc:3049] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:58:59.261296 14015 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:58:59.818148 14218 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:58:59.257304 14248 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:00.033998 14122 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:00.037979 14056 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.019s sys 0.016s
I20250113 08:59:00.023633 14219 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:00.237216 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.193850 14242 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:58:59.862165 14182 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:00.038292 12726 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:58:59.347146 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:00.251364 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 459) took 1016 ms. Trace:
I20250113 08:59:00.251725 12036 rpcz_store.cc:276] 0113 08:58:59.234926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:59.236176 (+ 1250us) service_pool.cc:225] Handling call
0113 08:59:00.251295 (+1015119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:00.037042 14084 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:00.252619 14084 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:58:59.891839 14120 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:00.117719 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:00.254722 11963 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:59:00.048050 14139 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:00.139376 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 86716us
I20250113 08:59:00.255635 14139 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.043365 14194 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:00.032488 14032 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:00.185106 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 158733us
I20250113 08:58:59.808971 14150 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:00.188477 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 156506us
W20250113 08:59:00.187050 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 925860us
W20250113 08:59:00.191411 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 335905us
I20250113 08:59:00.114778 14262 raft_consensus.cc:3049] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:00.026934 14245 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:58:59.415259 14206 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:00.054733 12721 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:00.276278 14139 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:00.276441 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 745) took 1671 ms (client timeout 1757 ms). Trace:
I20250113 08:59:00.115759 14263 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:00.277343 14267 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:59.874253 14152 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.280s
I20250113 08:59:00.120039 14208 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:00.035284 14025 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.218358 11859 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:00.038210 14141 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:00.128091 14244 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250113 08:59:00.186316 14205 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s user 0.010s sys 0.032s
I20250113 08:59:00.033113 14124 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.054085 14123 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:00.215044 14201 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.069s sys 0.060s
I20250113 08:59:00.333838 11892 raft_consensus.cc:1235] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:00.281577 14182 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:58:59.859804 14211 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:58:59.901703 14252 consensus_queue.cc:260] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.249712 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.343278 11893 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:00.253015 14084 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:00.270259 14194 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.265520 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.348204 11888 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:00.257624 12722 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:00.266491 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.266829 12725 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:00.266284 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.349795 11884 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:00.218217 14015 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:00.353286 14015 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:00.224622 14056 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.967s user 0.019s sys 0.017s
I20250113 08:59:00.356720 12723 raft_consensus.cc:1235] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:00.271268 12008 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:00.215610 14188 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:00.359159 14188 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:00.274827 14210 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:00.275957 14206 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.218663 14212 raft_consensus.cc:738] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:00.362820 14212 consensus_queue.cc:260] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:00.280509 11890 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:00.281441 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 89037.6us
I20250113 08:59:00.244338 14218 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:00.286252 12035 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:00.216396 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 185413us
I20250113 08:59:00.304157 14137 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:59:00.223135 14248 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:00.278662 14035 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:59:00.317826 14150 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:00.318670 14166 consensus_queue.cc:1046] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250113 08:59:00.320186 14264 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250113 08:59:00.320616 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.267356 14262 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:00.330891 11931 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:00.048348 14162 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:00.335505 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 476782us
I20250113 08:59:00.222203 14169 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.277858 14267 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:00.337548 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 119313us
I20250113 08:59:00.337601 14216 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:00.338163 14032 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:00.338742 14257 log.cc:926] Time spent T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.143s user 0.000s sys 0.000s
W20250113 08:59:00.276975 12721 rpcz_store.cc:269] 0113 08:58:58.604888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:58.605332 (+ 444us) service_pool.cc:225] Handling call
0113 08:59:00.276368 (+1671036us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75776}
W20250113 08:59:00.334244 14201 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.758s user 0.069s sys 0.060s
I20250113 08:59:00.331969 14161 raft_consensus.cc:3049] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:00.352694 14193 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:00.343021 14112 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:00.277580 14263 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.334152 14217 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.524186 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:00.529893 11859 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:00.424966 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 153752us
I20250113 08:59:00.524631 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:00.531560 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:00.439596 11932 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.717s (SENT) [suppressed 65 similar messages]
W20250113 08:59:00.446393 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 740) took 2992 ms (client timeout 2152 ms). Trace:
I20250113 08:59:00.371682 12723 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:59:00.532203 11963 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:00.531183 11859 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:00.534636 14165 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:00.586658 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 457) took 1507 ms. Trace:
W20250113 08:59:00.532711 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 93157.9us
W20250113 08:59:00.750416 14275 log.cc:926] Time spent T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20250113 08:59:00.531515 12722 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:00.458213 14258 log.cc:926] Time spent T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.345s user 0.000s sys 0.004s
I20250113 08:59:00.461994 14274 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:00.510018 14212 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:59:00.734126 14152 raft_consensus.cc:3049] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:00.642349 14025 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:00.348338 14167 consensus_queue.cc:1046] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.510s
W20250113 08:59:00.591899 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 321129us
I20250113 08:59:00.472110 14153 raft_consensus.cc:3049] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Stepping down as leader of term 2
W20250113 08:59:00.956410 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 425265us
I20250113 08:59:00.643585 11891 raft_consensus.cc:2388] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:00.523762 14246 consensus_queue.cc:1046] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.336s
I20250113 08:59:00.535953 12008 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:00.959720 11859 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:00.698962 14200 consensus_meta.cc:287] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.614s user 0.001s sys 0.010s
I20250113 08:59:00.860809 14112 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.715323 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:00.531683 14248 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:59:00.967551 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 431641us
I20250113 08:59:00.510437 14123 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:00.535537 12007 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:00.586488 14121 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:00.503136 14245 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:00.642045 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:00.364528 13989 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:59:00.370990 13950 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:00.608319 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.753855 14262 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:00.512943 14265 raft_consensus.cc:3049] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:59:00.648342 14141 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:00.572803 14193 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:00.642876 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.035393 11967 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:00.867110 12685 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:00.692184 12005 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.004s sys 0.037s
W20250113 08:59:01.037204 12005 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.199s user 0.008s sys 0.041s
W20250113 08:59:00.618261 12686 leader_election.cc:336] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.548s (SENT)
I20250113 08:59:00.980677 14032 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:59:00.533282 12725 rpcz_store.cc:269] 0113 08:58:57.447511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:57.646366 (+198855us) service_pool.cc:225] Handling call
0113 08:58:58.095324 (+448958us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:00.439834 (+2344510us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54007cd654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:00.440024 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4584844032}
I20250113 08:59:00.532073 11887 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:00.896477 12035 rpcz_store.cc:276] 0113 08:58:59.079408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:59.079575 (+ 167us) service_pool.cc:225] Handling call
0113 08:59:00.570076 (+1490501us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54003bc754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:00.586589 (+ 16513us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2636755328}
I20250113 08:59:00.877482 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:00.833905 14272 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
W20250113 08:59:00.617614 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 87669.6us
I20250113 08:59:00.643363 14210 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:00.957976 12006 connection.cc:232] RPC call timeout handler was delayed by -2.244420415000036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20250113 08:59:00.534801 14218 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:00.512573 14206 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:00.373454 14219 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:00.839062 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2090) took 1631 ms. Trace:
I20250113 08:59:00.530048 14161 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:00.647903 14162 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:00.646420 14169 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:00.643335 14190 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:00.970909 11885 raft_consensus.cc:1235] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:00.570025 14188 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:00.981369 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 445875us
W20250113 08:59:00.982195 11891 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1846) took 2688 ms (client timeout 1693 ms). Trace:
I20250113 08:59:01.288838 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:01.289644 12723 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:01.175949 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:00.940531 14274 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:01.295733 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.296530 12035 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:00.945955 14152 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:00.644416 14187 consensus_queue.cc:1046] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.330s
I20250113 08:59:00.344228 14182 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.007777 12722 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:01.286187 13989 tablet_bootstrap.cc:492] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Bootstrap complete.
I20250113 08:59:00.535780 14194 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:00.956478 14153 raft_consensus.cc:738] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:00.977810 14084 consensus_meta.cc:287] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.630s user 0.010s sys 0.002s
W20250113 08:59:00.978222 14015 consensus_meta.cc:287] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.624s user 0.010s sys 0.002s
W20250113 08:59:00.875247 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 544400us
I20250113 08:59:00.948945 14166 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20250113 08:59:00.990898 14124 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:01.002009 14122 consensus_meta.cc:287] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.754s user 0.013s sys 0.000s
I20250113 08:59:00.351213 14211 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.131904 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.292317 11858 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:01.036619 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 169994us
I20250113 08:59:01.323237 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:01.323585 12685 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:01.002244 14120 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:01.245527 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:01.215194 11967 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:01.291249 11891 rpcz_store.cc:269] 0113 08:58:58.282318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:58.423872 (+141554us) service_pool.cc:225] Handling call
0113 08:59:00.970368 (+2546496us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b540038f254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:00.970549 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4939228928}
I20250113 08:59:01.324651 11889 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:01.206465 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 246803us
I20250113 08:59:01.323926 12040 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:01.289959 12006 connection.cc:232] RPC call timeout handler was delayed by -2.1818979140000465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
I20250113 08:59:01.294040 12042 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:59:01.038579 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 420334us
I20250113 08:59:01.002466 14196 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.294659 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 447) took 1010 ms. Trace:
I20250113 08:59:01.286586 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:01.327564 11887 rpcz_store.cc:276] 0113 08:59:00.284566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:00.284716 (+ 150us) service_pool.cc:225] Handling call
0113 08:59:01.294596 (+1009880us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:01.298372 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:01.328547 12037 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:59:01.326979 12006 leader_election.cc:336] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.617s (SENT)
I20250113 08:59:01.325582 11859 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:01.329293 12006 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:01.283520 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 511) took 1247 ms. Trace:
I20250113 08:59:01.300851 12038 raft_consensus.cc:1235] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:01.300949 14263 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:01.299677 14274 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:01.259044 14209 consensus_meta.cc:287] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.213s user 0.003s sys 0.011s
I20250113 08:59:01.292481 14262 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.302454 14161 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:01.093565 14208 consensus_meta.cc:287] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.784s user 0.009s sys 0.005s
I20250113 08:59:01.305202 11934 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:01.305689 14273 consensus_queue.cc:1159] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 3 } }
I20250113 08:59:01.294811 14218 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.291469 14137 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.310886 14182 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:01.311205 14187 raft_consensus.cc:3049] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:01.282963 14271 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:01.093969 14245 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:01.035228 14013 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.018693 14265 raft_consensus.cc:738] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:01.314147 12035 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:01.284206 14193 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:01.358879 14112 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:01.121740 14216 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:01.291396 14279 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.375020 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.317485 12723 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:01.393669 14245 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:01.402719 14245 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:01.319015 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:00.994475 14123 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:01.329370 12685 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:01.299468 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:01.423777 12725 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
W20250113 08:59:01.317167 14281 log.cc:926] Time spent T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Append to log took a long time: real 0.452s user 0.002s sys 0.000s
I20250113 08:59:01.330592 12007 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:01.391654 14220 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:01.427960 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 97441.8us
W20250113 08:59:01.334815 12006 leader_election.cc:336] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.646s (SENT)
I20250113 08:59:01.428889 12006 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:01.338189 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 434) took 2476 ms. Trace:
I20250113 08:59:01.429872 11885 rpcz_store.cc:276] 0113 08:58:58.852854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:58.854455 (+ 1601us) service_pool.cc:225] Handling call
0113 08:59:00.970673 (+2116218us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:01.291617 (+320944us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:01.328871 (+ 37254us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b540038f250. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:01.328997 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4865771264}
W20250113 08:59:01.338415 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 46194us
W20250113 08:59:01.339316 11839 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 08aa17121f5b429b8a32299911c7ef76, TS 6ea00811b7d44b49b872ac5c03b91487 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.341648 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:01.432415 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 08aa17121f5b429b8a32299911c7ef76, TS 6ea00811b7d44b49b872ac5c03b91487 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.432718 12037 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:59:01.338220 11859 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:01.434123 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 95987.3us
I20250113 08:59:01.315193 14084 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.327159 13989 ts_tablet_manager.cc:1397] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent bootstrapping tablet: real 4.593s user 0.060s sys 0.065s
I20250113 08:59:01.446766 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.346822 14153 consensus_queue.cc:260] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.357730 14209 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.358335 14252 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:59:01.327847 14219 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.298889 14210 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.313663 14165 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250113 08:59:01.326579 14120 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:01.370525 11963 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:01.342259 14235 consensus_meta.cc:287] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.155s user 0.005s sys 0.010s
I20250113 08:59:01.376478 14208 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:01.335392 12717 rpcz_store.cc:276] 0113 08:59:00.035657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:00.037111 (+ 1454us) service_pool.cc:225] Handling call
0113 08:59:00.121821 (+ 84710us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:00.249061 (+127240us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:00.258159 (+ 9098us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:00.316390 (+ 58231us) log.cc:844] Serialized 26 byte log entry
0113 08:59:00.723987 (+407597us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:00.724069 (+ 82us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:00.724190 (+ 121us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:01.250699 (+526509us) raft_consensus.cc:1680] finished
0113 08:59:01.257050 (+ 6351us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:01.283448 (+ 26398us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":632,"raft.queue_time_us":21815,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":136,"thread_start_us":406941,"threads_started":1,"wal-append.queue_time_us":672371}
W20250113 08:59:01.377600 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 72490.5us
I20250113 08:59:01.466074 11934 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:01.378204 14206 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:01.339704 12721 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:01.381453 14137 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.382818 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.385428 11889 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:01.480839 12038 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:01.481588 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.338572 14277 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:59:01.330792 14032 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:01.369527 14232 consensus_meta.cc:287] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.177s user 0.011s sys 0.004s
I20250113 08:59:01.347393 14213 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:01.361322 14167 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:01.485577 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.486215 11884 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:01.317065 14015 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.001727 14121 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.493018 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.399691 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.493695 12035 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:59:01.393491 14242 consensus_meta.cc:287] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.143s user 0.013s sys 0.000s
I20250113 08:59:01.320062 14122 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.392138 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 648) took 2240 ms. Trace:
I20250113 08:59:01.396893 14013 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.420917 12008 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:01.386451 14187 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:01.389318 14271 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:01.425310 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 99476.8us
I20250113 08:59:01.426224 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.330044 12724 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:01.342187 14152 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.439443 13950 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:01.483922 14206 raft_consensus.cc:695] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:01.441627 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.443173 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.396935 14216 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:01.397217 14161 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.339932 14196 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.471489 14235 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.456128 11885 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:01.514683 12685 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:01.464124 14211 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:01.468868 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.537434 11891 raft_consensus.cc:2363] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4.
I20250113 08:59:01.469854 14084 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.470641 14153 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.450484 14291 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:01.473141 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:01.537912 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.547009 12717 raft_consensus.cc:2371] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3. Re-sending same reply.
I20250113 08:59:01.546645 14187 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.548007 14187 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.548274 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:01.458964 14212 consensus_meta.cc:287] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.504s user 0.009s sys 0.003s
I20250113 08:59:01.474963 12723 raft_consensus.cc:1235] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:01.550092 14294 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250113 08:59:01.550444 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.477593 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.550858 14137 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:01.478713 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.477860 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.551889 11962 raft_consensus.cc:2388] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:01.552512 14285 log.cc:926] Time spent T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20250113 08:59:01.552654 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.552716 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.553974 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.554047 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.492282 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.515131 14263 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:01.554811 11964 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:01.555116 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.508388 14297 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20250113 08:59:01.555683 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.462838 14120 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.556252 11964 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:01.558041 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:01.558718 14248 consensus_meta.cc:287] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.557s user 0.003s sys 0.007s
I20250113 08:59:01.559267 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:01.559195 14248 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.400943 14279 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.560583 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:01.257608 14267 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.493790 11886 raft_consensus.cc:2388] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:01.561861 12718 raft_consensus.cc:1235] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:01.502513 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.562963 14248 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:01.563346 14293 log.cc:926] Time spent T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250113 08:59:01.563421 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.564304 11966 raft_consensus.cc:2388] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 2: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:01.289958 11959 rpcz_store.cc:276] 0113 08:58:59.208141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:59.216405 (+ 8264us) service_pool.cc:225] Handling call
0113 08:58:59.228785 (+ 12380us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:58:59.229273 (+ 488us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:59.229309 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:58:59.229920 (+ 611us) log.cc:844] Serialized 26 byte log entry
0113 08:58:59.777618 (+547698us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:58:59.795708 (+ 18090us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:58:59.819940 (+ 24232us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:00.822285 (+1002345us) raft_consensus.cc:1680] finished
0113 08:59:00.822322 (+ 37us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:00.838967 (+ 16645us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5480,"raft.queue_time_us":1711,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"thread_start_us":459747,"threads_started":1,"wal-append.queue_time_us":822519}
I20250113 08:59:01.504772 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:01.505036 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 100939us
I20250113 08:59:01.474126 14169 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:01.557024 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.567451 11964 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:59:01.454787 14274 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.531997 13950 raft_consensus.cc:695] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:01.574389 14279 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:01.574491 13950 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.576366 14121 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:01.578123 14015 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.578696 14013 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.579476 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.579916 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.580655 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.539680 14182 consensus_queue.cc:1046] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20250113 08:59:01.568778 14084 consensus_queue.cc:1046] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20250113 08:59:01.581950 14182 raft_consensus.cc:3049] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:01.582353 14182 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:01.583324 14182 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.494529 14232 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:01.590728 14286 log.cc:926] Time spent T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20250113 08:59:01.592223 14291 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:01.592581 14232 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:01.503415 12720 rpcz_store.cc:276] 0113 08:58:59.148642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:59.222580 (+ 73938us) service_pool.cc:225] Handling call
0113 08:58:59.228970 (+ 6390us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:58:59.844261 (+615291us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:58:59.853400 (+ 9139us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:58:59.946648 (+ 93248us) log.cc:844] Serialized 26 byte log entry
0113 08:59:00.265890 (+319242us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:00.265990 (+ 100us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:00.276943 (+ 10953us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:01.362306 (+1085363us) raft_consensus.cc:1680] finished
0113 08:59:01.367310 (+ 5004us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:01.386305 (+ 18995us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14860,"raft.queue_time_us":611501,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":6422,"thread_start_us":232849,"threads_started":1,"wal-append.queue_time_us":327983}
I20250113 08:59:01.593441 14291 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.594878 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.596684 14235 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:01.537273 14216 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.568239 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.601930 12684 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:01.603149 12684 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:01.500150 12035 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:01.505920 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.510838 12039 raft_consensus.cc:1235] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:01.511386 11934 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:01.456024 14218 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.584789 14182 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.606344 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.606691 14122 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.512984 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.607486 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.608311 11884 raft_consensus.cc:2388] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:01.608250 11964 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:01.424024 12036 raft_consensus.cc:2371] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3. Re-sending same reply.
I20250113 08:59:01.449384 14123 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:01.528193 12037 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:01.508140 14271 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.528785 12725 raft_consensus.cc:2371] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4. Re-sending same reply.
I20250113 08:59:01.612051 12725 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:01.528690 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 234554us
I20250113 08:59:01.612581 12684 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:01.462617 14292 consensus_queue.cc:1046] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20250113 08:59:01.514703 12041 raft_consensus.cc:1235] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:01.527864 11890 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:59:01.519600 12686 leader_election.cc:336] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.917s (SENT)
I20250113 08:59:01.515494 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.497336 14208 raft_consensus.cc:2743] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:59:01.534880 12685 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:01.541867 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:01.508396 14277 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:01.510914 14290 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.615069 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:01.544977 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:01.550243 14206 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.555833 11962 raft_consensus.cc:2388] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:01.668275 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.485363 14032 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.485467 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.565032 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.670750 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.672173 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:01.571662 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 236881us
I20250113 08:59:01.461653 14265 consensus_queue.cc:260] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.539678 14201 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:01.675781 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.677389 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.678076 12725 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.607553 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:01.609794 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 98521.3us
W20250113 08:59:01.643373 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 108593us
I20250113 08:59:01.569671 14169 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:01.651532 12684 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:01.643664 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 124114us
I20250113 08:59:01.654482 14193 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:01.611241 13989 raft_consensus.cc:357] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:01.658670 14274 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.659358 12722 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:56200
I20250113 08:59:01.660295 14120 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.661628 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.689400 14169 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:01.510890 14162 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.638753 14248 consensus_queue.cc:1046] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20250113 08:59:01.661554 14112 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:01.671885 14213 raft_consensus.cc:695] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:01.667281 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.703132 14277 raft_consensus.cc:695] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:01.669293 12035 raft_consensus.cc:2363] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
I20250113 08:59:01.672942 11966 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:59:01.673542 12006 leader_election.cc:336] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.693s (SENT)
I20250113 08:59:01.668447 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.670140 11892 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:01.590801 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.612150 14232 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:01.674798 11886 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:01.611896 14121 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:59:01.683511 12686 leader_election.cc:336] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.022s (SENT)
I20250113 08:59:01.685103 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.620103 14267 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:01.611205 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:01.690972 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:01.619493 14211 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.692418 11960 raft_consensus.cc:1235] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:01.644949 14015 raft_consensus.cc:3049] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:01.693820 12684 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:01.707018 11884 raft_consensus.cc:1235] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:01.706516 12037 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:01.706044 12033 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:59:01.662792 12038 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:01.713373 11891 raft_consensus.cc:1235] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 2. Ops: []
I20250113 08:59:01.682575 14167 raft_consensus.cc:695] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:01.718816 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.714998 12725 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:01.607631 14218 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.693930 14180 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:01.684713 14201 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:01.493172 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.694270 13989 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Initialized, Role: LEARNER
I20250113 08:59:01.694833 14209 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.670053 14124 consensus_queue.cc:1046] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20250113 08:59:01.682618 14288 consensus_queue.cc:1046] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20250113 08:59:01.709806 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.693552 14166 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20250113 08:59:01.670629 14290 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.724277 12034 raft_consensus.cc:1235] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:01.910100 12006 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:01.911283 12686 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:01.912554 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 229082us
I20250113 08:59:01.913125 11964 raft_consensus.cc:2388] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:01.914564 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:01.915391 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.915962 11968 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.916747 12724 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 1 FOLLOWER]: Advancing to term 3
I20250113 08:59:01.963096 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.963948 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:01.965994 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 292452us
W20250113 08:59:01.964015 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 270268us
W20250113 08:59:01.966717 12006 leader_election.cc:336] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.699s (SENT)
I20250113 08:59:01.967229 12006 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:01.967496 11859 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:01.967698 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:01.968976 14208 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.913544 14015 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:01.969601 14208 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:01.970157 14015 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.971532 14015 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.971292 14169 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.972312 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.966521 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.971627 14208 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:01.962749 14267 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:01.974300 11858 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:01.978536 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:01.979288 11965 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:01.979864 11858 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:01.980098 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:01.898699 14284 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.652s user 0.000s sys 0.003s
I20250113 08:59:01.975451 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:01.964756 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:01.919562 12008 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:01.911134 14120 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:01.977787 12684 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:01.973212 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:01.981631 11964 raft_consensus.cc:2388] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:01.982283 11965 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:01.965130 11961 raft_consensus.cc:2405] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250113 08:59:01.982311 11962 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:59:01.981034 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:01.721863 14112 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:02.063210 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.064692 12038 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:02.008615 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:02.065486 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 98796.7us
I20250113 08:59:01.974922 12717 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:02.010866 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 91385.3us
I20250113 08:59:02.013152 12037 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:59:01.912564 14121 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:02.071538 11968 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:02.074131 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:02.074584 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 161236us
I20250113 08:59:02.000331 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:02.075278 12686 leader_election.cc:336] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.161s (SENT)
I20250113 08:59:02.075748 12686 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:02.074648 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:02.076696 12037 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:01.966333 11966 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:59:02.066026 12006 connection.cc:232] RPC call timeout handler was delayed by -2.0599202120000655s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1786s for a response
W20250113 08:59:02.081867 12006 connection.cc:232] RPC call timeout handler was delayed by -1.9690293799999381s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20250113 08:59:02.082829 12006 connection.cc:232] RPC call timeout handler was delayed by -1.5727796600000374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
W20250113 08:59:02.083256 12006 connection.cc:232] RPC call timeout handler was delayed by -1.2523796600000878s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20250113 08:59:01.966722 14302 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.976321 14232 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:01.967005 14201 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:01.998507 14137 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:01.740430 14296 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:02.085917 12006 leader_election.cc:336] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.501s (SENT)
I20250113 08:59:02.074443 11959 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:02.098881 12006 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:02.100550 12006 leader_election.cc:336] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.516s (SENT)
W20250113 08:59:02.101742 12006 leader_election.cc:336] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.770s (SENT)
I20250113 08:59:02.101868 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.102360 12006 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:02.103714 14305 consensus_queue.cc:1046] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20250113 08:59:01.999056 14193 raft_consensus.cc:695] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:02.087694 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 403476us
I20250113 08:59:02.011492 14150 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:01.975020 12036 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:01.721617 14162 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:01.999727 14267 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:02.092976 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.073540 14306 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:59:02.097829 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:01.976055 14211 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.081040 12724 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
W20250113 08:59:02.107781 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 189631us
I20250113 08:59:02.110169 14248 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:02.073664 14209 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:02.111303 12038 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:59:02.112761 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 319434us
I20250113 08:59:02.087278 14121 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:02.160948 12725 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:02.115567 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.074016 14208 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:02.162057 11891 raft_consensus.cc:2388] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 2: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:02.074671 14169 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:02.118455 11932 leader_election.cc:400] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4.
I20250113 08:59:02.164794 11932 leader_election.cc:403] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:02.075064 14123 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:02.120549 12037 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:59:01.978294 14274 raft_consensus.cc:2743] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:59:02.114856 14013 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:02.119138 14124 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:02.119151 14150 raft_consensus.cc:695] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:02.095841 14232 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:02.067250 14112 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:02.119390 14218 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:02.127696 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.127920 14308 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election won for term 3
W20250113 08:59:02.149808 12006 leader_election.cc:336] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.777s (SENT)
I20250113 08:59:02.178400 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.183772 12006 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:02.184191 11891 raft_consensus.cc:2388] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:59:02.122602 12721 consensus_meta.cc:287] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.628s user 0.002s sys 0.008s
I20250113 08:59:02.151141 12008 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:02.155453 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:02.159571 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.185360 12038 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:02.123351 14153 raft_consensus.cc:2743] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250113 08:59:02.165836 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 58113.2us
W20250113 08:59:02.188910 12006 leader_election.cc:336] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.786s (SENT)
I20250113 08:59:02.189894 11932 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:02.106705 14137 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.164234 14209 raft_consensus.cc:695] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:02.070775 13989 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:02.181731 11884 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:02.184005 12037 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2.
I20250113 08:59:02.154906 14248 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:02.094844 14302 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:02.123363 14296 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:02.193122 12006 leader_election.cc:336] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.857s (SENT)
I20250113 08:59:02.175698 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:02.187724 14262 consensus_meta.cc:287] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.820s user 0.004s sys 0.007s
I20250113 08:59:02.120488 14167 consensus_queue.cc:237] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.123113 14213 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.120074 14277 consensus_queue.cc:237] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.177032 14124 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:02.186743 12721 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:02.711652 14152 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:02.712093 12720 raft_consensus.cc:1235] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:02.271433 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:02.713016 12724 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:02.694952 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.714092 14265 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:59:02.714497 12035 raft_consensus.cc:2388] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:02.714953 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2141) took 1132 ms. Trace:
I20250113 08:59:02.190037 14169 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4.
W20250113 08:59:02.606539 11890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 450) took 1431 ms (client timeout 1606 ms). Trace:
I20250113 08:59:02.308013 14301 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:59:02.716135 11890 rpcz_store.cc:269] 0113 08:59:01.175472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.175615 (+ 143us) service_pool.cc:225] Handling call
0113 08:59:02.606270 (+1430655us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54000d7d54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.606447 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":544646912}
I20250113 08:59:02.289305 14150 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.537726 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 651) took 2503 ms. Trace:
I20250113 08:59:02.717803 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.718710 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1868) took 1001 ms. Trace:
I20250113 08:59:02.717729 12719 rpcz_store.cc:276] 0113 08:59:00.034309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:00.047999 (+ 13690us) service_pool.cc:225] Handling call
0113 08:59:00.112757 (+ 64758us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:01.352675 (+1239918us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:01.353021 (+ 346us) log.cc:844] Serialized 14 byte log entry
0113 08:59:01.395685 (+ 42664us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:01.398220 (+ 2535us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:01.423592 (+ 25372us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.537457 (+1113865us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54006d06d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:02.537515 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:02.537653 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4547,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":2233,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":8273,"spinlock_wait_cycles":2840110464,"thread_start_us":33673,"threads_started":1,"wal-append.queue_time_us":41440,"wal-append.run_cpu_time_us":1183,"wal-append.run_wall_time_us":1006943}
W20250113 08:59:02.447208 14310 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20250113 08:59:02.406492 14206 consensus_queue.cc:1046] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.314s
I20250113 08:59:02.720310 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:02.421124 12717 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:02.534597 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 464) took 1245 ms. Trace:
I20250113 08:59:02.715287 12035 rpcz_store.cc:276] 0113 08:59:01.582323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.694570 (+1112247us) service_pool.cc:225] Handling call
0113 08:59:02.714873 (+ 20303us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.721314 14306 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:02.721566 12039 rpcz_store.cc:276] 0113 08:59:01.288783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.289002 (+ 219us) service_pool.cc:225] Handling call
0113 08:59:01.298428 (+ 9426us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:01.605564 (+307136us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.534336 (+928772us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b540002d5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:02.534533 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":485076480}
I20250113 08:59:02.722579 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:02.723250 12035 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:02.720715 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 751) took 1269 ms. Trace:
I20250113 08:59:02.270253 14303 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:02.421651 14307 log.cc:926] Time spent T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.266s user 0.001s sys 0.000s
I20250113 08:59:02.723929 12720 rpcz_store.cc:276] 0113 08:59:01.451349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.596108 (+144759us) service_pool.cc:225] Handling call
0113 08:59:02.711891 (+1115783us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:02.712703 (+ 812us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.720520 (+ 7817us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54006e1750. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.720654 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2564502400}
I20250113 08:59:02.427151 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 469) took 1119 ms. Trace:
I20250113 08:59:02.725639 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:02.725554 12034 rpcz_store.cc:276] 0113 08:59:01.314748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.316563 (+ 1815us) service_pool.cc:225] Handling call
0113 08:59:01.316926 (+ 363us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:01.909705 (+592779us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.426863 (+517158us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b54000cd0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:02.427054 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":931758464}
I20250113 08:59:02.432006 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.192219 14219 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:02.727698 14219 raft_consensus.cc:695] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:02.728659 14219 consensus_queue.cc:237] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:02.343432 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 363607us
I20250113 08:59:02.729538 11967 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
I20250113 08:59:02.730381 11858 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:02.730574 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 375) took 1223 ms (client timeout 1594 ms). Trace:
W20250113 08:59:02.730978 11967 rpcz_store.cc:269] 0113 08:59:01.507467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.553673 (+ 46206us) service_pool.cc:225] Handling call
0113 08:59:02.730408 (+1176735us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540028dcd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.730500 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2702077824}
I20250113 08:59:02.270442 14315 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:02.731793 11893 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:02.603185 14242 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:02.733428 14013 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:02.671411 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.735039 12040 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:02.688181 14121 consensus_meta.cc:287] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.519s user 0.010s sys 0.000s
W20250113 08:59:02.577618 14032 consensus_meta.cc:287] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.884s user 0.008s sys 0.004s
I20250113 08:59:02.736037 14121 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.554514 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 506) took 3795 ms. Trace:
I20250113 08:59:02.738593 12720 raft_consensus.cc:2388] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:59:02.594753 14161 consensus_meta.cc:287] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.056s user 0.011s sys 0.002s
I20250113 08:59:02.739066 11962 raft_consensus.cc:1270] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250113 08:59:02.703771 11885 raft_consensus.cc:3049] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:02.740338 11885 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:02.741082 11885 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.742182 11885 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:02.734795 14013 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:02.604244 14220 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:02.743271 14220 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:02.707484 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 729706us
I20250113 08:59:02.425315 14211 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:02.745436 14141 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:02.537770 11886 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:59:02.424947 14187 consensus_meta.cc:287] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.876s user 0.011s sys 0.000s
I20250113 08:59:02.747465 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1867) took 1055 ms. Trace:
I20250113 08:59:02.282085 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.747867 11886 rpcz_store.cc:276] 0113 08:59:01.691973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.916263 (+224290us) service_pool.cc:225] Handling call
0113 08:59:02.747320 (+831057us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b540034d0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.747414 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":194297088}
I20250113 08:59:02.737553 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1696) took 1264 ms. Trace:
I20250113 08:59:02.748916 11893 rpcz_store.cc:276] 0113 08:59:01.472742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.472891 (+ 149us) service_pool.cc:225] Handling call
0113 08:59:02.737401 (+1264510us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54000609d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.737497 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2712842496}
I20250113 08:59:02.750082 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.751111 11963 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:59:02.706893 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 631589us
I20250113 08:59:02.751716 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:02.459627 14287 log.cc:926] Time spent T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 1.167s user 0.001s sys 0.008s
I20250113 08:59:02.733387 12006 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:02.669879 14112 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.539398 14246 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:02.753204 12723 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:59:02.550904 14271 consensus_meta.cc:287] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.940s user 0.011s sys 0.003s
W20250113 08:59:02.754380 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 561281us
I20250113 08:59:02.754057 14246 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.754865 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 757) took 1289 ms. Trace:
W20250113 08:59:02.755088 12006 leader_election.cc:336] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.859s (SENT)
I20250113 08:59:02.754655 14271 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.755297 12723 rpcz_store.cc:276] 0113 08:59:01.465818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.550166 (+ 84348us) service_pool.cc:225] Handling call
0113 08:59:02.754647 (+1204481us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400771ad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.754743 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2750586624}
W20250113 08:59:02.755900 12006 leader_election.cc:336] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.866s (SENT)
I20250113 08:59:02.756156 11888 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:02.756325 12006 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:02.340643 14245 consensus_meta.cc:287] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.937s user 0.012s sys 0.000s
I20250113 08:59:02.745883 14013 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:02.757370 14245 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.757611 14013 consensus_queue.cc:1046] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:02.757993 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1695) took 1286 ms. Trace:
I20250113 08:59:02.758617 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:02.758394 11888 rpcz_store.cc:276] 0113 08:59:01.471734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.471887 (+ 153us) service_pool.cc:225] Handling call
0113 08:59:01.472255 (+ 368us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:02.756615 (+1284360us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.757749 (+ 1134us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540026cd54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:02.757899 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2952319744}
I20250113 08:59:02.758859 14013 raft_consensus.cc:3049] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:02.759390 14013 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:02.753674 14015 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:02.759735 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.703565 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.760063 14245 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:02.760430 11884 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:02.760536 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 456) took 1046 ms. Trace:
I20250113 08:59:02.168778 14314 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:02.752161 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 367) took 1274 ms. Trace:
I20250113 08:59:02.760938 11888 rpcz_store.cc:276] 0113 08:59:01.713820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.759540 (+1045720us) service_pool.cc:225] Handling call
0113 08:59:02.760483 (+ 943us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.761273 11963 rpcz_store.cc:276] 0113 08:59:01.477265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.477375 (+ 110us) service_pool.cc:225] Handling call
0113 08:59:02.752006 (+1274631us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400050ad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.752103 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2732908288}
I20250113 08:59:02.762372 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2142) took 1179 ms. Trace:
I20250113 08:59:02.762323 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:02.430670 14216 consensus_meta.cc:287] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.833s user 0.006s sys 0.004s
I20250113 08:59:02.638200 14194 raft_consensus.cc:3049] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:02.762810 12037 rpcz_store.cc:276] 0113 08:59:01.582643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.092621 (+509978us) service_pool.cc:225] Handling call
0113 08:59:02.762309 (+669688us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.770023 12684 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:02.772050 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.189069 14193 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.741575 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 756) took 1278 ms. Trace:
W20250113 08:59:02.634663 14182 consensus_meta.cc:287] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.019s user 0.010s sys 0.003s
I20250113 08:59:02.763737 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.190980 14274 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:02.748298 11891 raft_consensus.cc:2371] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6. Re-sending same reply.
I20250113 08:59:02.789004 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1865) took 1097 ms. Trace:
I20250113 08:59:02.789549 11891 rpcz_store.cc:276] 0113 08:59:01.691420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.190252 (+498832us) service_pool.cc:225] Handling call
0113 08:59:02.788937 (+598685us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.764411 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:02.709163 14180 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.004s sys 0.008s
I20250113 08:59:02.356268 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 461) took 2538 ms. Trace:
I20250113 08:59:02.757055 12723 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250113 08:59:02.765086 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.724421 14303 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:02.713771 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 518) took 1693 ms (client timeout 2003 ms). Trace:
I20250113 08:59:02.766458 11962 raft_consensus.cc:1270] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250113 08:59:02.767160 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.709973 11960 raft_consensus.cc:1235] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:02.756707 14246 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:02.290216 11932 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:02.769678 11859 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:02.703917 14263 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:02.905756 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.773833 11963 raft_consensus.cc:1235] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:02.472637 12725 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
W20250113 08:59:02.426088 14282 log.cc:926] Time spent T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 1.142s user 0.008s sys 0.000s
I20250113 08:59:02.719107 11890 rpcz_store.cc:276] 0113 08:59:01.717766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.717548 (+999782us) service_pool.cc:225] Handling call
0113 08:59:02.718653 (+ 1105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.917394 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.918267 11890 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:02.919332 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.920269 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.775411 11966 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:02.776196 11885 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250113 08:59:02.779361 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:02.779659 12724 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:02.924172 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:02.708344 14210 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:02.191080 14252 consensus_meta.cc:287] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.732s user 0.005s sys 0.006s
I20250113 08:59:02.736667 14032 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.392093 12038 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:02.788033 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1869) took 1041 ms. Trace:
I20250113 08:59:02.948180 11888 rpcz_store.cc:276] 0113 08:59:01.746796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.763543 (+1016747us) service_pool.cc:225] Handling call
0113 08:59:02.787972 (+ 24429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.950284 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 452) took 1265 ms. Trace:
I20250113 08:59:02.950752 11888 rpcz_store.cc:276] 0113 08:59:01.684753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.949120 (+1264367us) service_pool.cc:225] Handling call
0113 08:59:02.949563 (+ 443us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:02.949819 (+ 256us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:02.949858 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:02.949916 (+ 58us) raft_consensus.cc:1640] Marking committed up to 9
0113 08:59:02.949990 (+ 74us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.950055 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:02.950194 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.709679 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2130) took 1311 ms. Trace:
I20250113 08:59:02.952383 12033 rpcz_store.cc:276] 0113 08:59:01.398465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.425887 (+ 27422us) service_pool.cc:225] Handling call
0113 08:59:02.623577 (+1197690us) spinlock_profiling.cc:243] Waited 54.9 ms on lock 0x7b54000cc6d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:02.709600 (+ 86023us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126281088}
I20250113 08:59:02.953699 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:02.723201 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.956233 12039 raft_consensus.cc:2388] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:02.956775 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1591) took 1280 ms. Trace:
I20250113 08:59:02.957170 12039 rpcz_store.cc:276] 0113 08:59:01.676882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.722976 (+1046094us) service_pool.cc:225] Handling call
0113 08:59:02.956696 (+233720us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.459475 13989 ts_tablet_manager.cc:1428] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent starting tablet: real 1.014s user 0.007s sys 0.033s
I20250113 08:59:02.188854 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:02.783113 14273 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:02.797434 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:02.993274 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1600) took 1232 ms. Trace:
I20250113 08:59:02.993760 12034 rpcz_store.cc:276] 0113 08:59:01.760457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.765784 (+1005327us) service_pool.cc:225] Handling call
0113 08:59:02.993150 (+227366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:02.428735 14166 raft_consensus.cc:3049] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:02.706602 11959 raft_consensus.cc:1270] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250113 08:59:02.732811 14242 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:02.815017 12042 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:02.851250 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2143) took 1268 ms. Trace:
I20250113 08:59:02.792063 11856 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:02.224327 14311 raft_consensus.cc:3049] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:02.899405 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1871) took 1046 ms. Trace:
I20250113 08:59:03.055595 11886 rpcz_store.cc:276] 0113 08:59:01.852746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.766906 (+914160us) service_pool.cc:225] Handling call
0113 08:59:02.899315 (+132409us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.017066 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:02.804812 11967 raft_consensus.cc:1270] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250113 08:59:02.224323 14248 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:02.902954 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 520) took 1437 ms. Trace:
I20250113 08:59:02.898921 12041 rpcz_store.cc:276] 0113 08:58:59.801608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:59.801739 (+ 131us) service_pool.cc:225] Handling call
0113 08:58:59.802112 (+ 373us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:01.644392 (+1842280us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.339924 (+695532us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54002b8c54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:02.340119 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3938163072}
W20250113 08:59:03.057885 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 305832us
I20250113 08:59:03.057703 12723 rpcz_store.cc:276] 0113 08:59:01.465557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.756488 (+1290931us) service_pool.cc:225] Handling call
0113 08:59:02.756740 (+ 252us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:02.902647 (+145907us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.902890 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.058528 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:02.771098 12006 leader_election.cc:336] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.950s (SENT)
I20250113 08:59:03.059566 12006 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:02.783643 14216 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:02.721354 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 661) took 1151 ms (client timeout 1513 ms). Trace:
I20250113 08:59:02.429927 14137 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:02.340461 14308 raft_consensus.cc:695] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:02.358471 14290 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:02.771639 14162 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:03.005631 14112 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:02.926380 11934 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:03.075654 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 304561us
I20250113 08:59:03.047191 14206 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:59:03.078362 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 152095us
W20250113 08:59:03.078958 12006 leader_election.cc:336] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.956s (SENT)
W20250113 08:59:02.706252 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.24495e+06us
I20250113 08:59:02.923173 11965 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:59:03.079962 12006 leader_election.cc:336] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.116s (SENT)
I20250113 08:59:02.922513 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1703) took 1256 ms. Trace:
I20250113 08:59:03.080338 12006 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:02.781730 14194 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:02.931169 11839 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e2677547aaee4bba86cf339fe13278bf, TS 37bd4bed3ee6423c8dac0cef3bf6882a -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:03.085834 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e2677547aaee4bba86cf339fe13278bf, TS 37bd4bed3ee6423c8dac0cef3bf6882a -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:02.936977 12037 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:03.087214 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 667) took 1392 ms (client timeout 1661 ms). Trace:
W20250113 08:59:03.087587 12725 rpcz_store.cc:269] 0113 08:59:01.694341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.914252 (+219911us) service_pool.cc:225] Handling call
0113 08:59:03.087053 (+1172801us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b5400501254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:03.087157 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":564862336}
W20250113 08:59:02.939101 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 169498us
I20250113 08:59:02.760838 14245 consensus_queue.cc:1046] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.146s
I20250113 08:59:03.063045 14124 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:02.850492 12721 rpcz_store.cc:269] 0113 08:59:01.020648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.118994 (+ 98346us) service_pool.cc:225] Handling call
0113 08:59:02.713692 (+1594698us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:02.899866 14180 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.886s user 0.012s sys 0.011s
I20250113 08:59:02.784327 14211 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:02.964454 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 474) took 1197 ms. Trace:
I20250113 08:59:03.100553 12033 rpcz_store.cc:276] 0113 08:59:01.767559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.953381 (+1185822us) service_pool.cc:225] Handling call
0113 08:59:02.964375 (+ 10994us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.102128 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:02.995328 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.917644 14263 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:03.006999 11968 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.022s sys 0.051s
I20250113 08:59:03.106588 12034 raft_consensus.cc:2363] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
W20250113 08:59:03.106638 11968 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.022s sys 0.051s
I20250113 08:59:03.107159 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1584) took 1473 ms. Trace:
I20250113 08:59:03.107609 12034 rpcz_store.cc:276] 0113 08:59:01.634100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.994941 (+1360841us) service_pool.cc:225] Handling call
0113 08:59:03.107102 (+112161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.109496 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.057844 14262 consensus_queue.cc:1035] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250113 08:59:03.084204 14141 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:59:03.115738 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.052297 12035 rpcz_store.cc:276] 0113 08:59:01.582986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.764165 (+1181179us) service_pool.cc:225] Handling call
0113 08:59:02.851167 (+ 87002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.076207 14196 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.054402 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:02.909195 11884 raft_consensus.cc:2388] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:03.059154 12041 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
W20250113 08:59:03.062381 12686 connection.cc:232] RPC call timeout handler was delayed by -1.2765611369999306s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2922s for a response
W20250113 08:59:03.062216 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 772063us
I20250113 08:59:03.120759 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1592) took 1426 ms. Trace:
W20250113 08:59:02.966532 14201 consensus_meta.cc:287] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.852s user 0.004s sys 0.009s
W20250113 08:59:03.120728 12686 connection.cc:232] RPC call timeout handler was delayed by -1.1212117269999453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
I20250113 08:59:03.121171 12041 rpcz_store.cc:276] 0113 08:59:01.694596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.058248 (+1363652us) service_pool.cc:225] Handling call
0113 08:59:03.120695 (+ 62447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.122237 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:02.869494 14165 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.144s
I20250113 08:59:02.761334 14314 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:03.086172 14206 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:03.065639 11931 leader_election.cc:400] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
W20250113 08:59:03.135318 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 69922.9us
I20250113 08:59:03.136107 11931 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:02.975090 14129 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:02.760744 14013 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:03.068032 14297 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20250113 08:59:03.012363 14288 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.043s sys 0.090s
I20250113 08:59:02.407732 14209 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:02.969731 13989 ts_tablet_manager.cc:1460] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Tablet startup took 1524ms
I20250113 08:59:02.716444 14301 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:03.064811 14188 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:03.079030 11887 raft_consensus.cc:1235] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:03.080534 11885 rpcz_store.cc:276] 0113 08:59:01.665957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.666090 (+ 133us) service_pool.cc:225] Handling call
0113 08:59:01.674966 (+ 8876us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:02.453564 (+778598us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b5400169f20. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336599990 00007fc33660b300 00007fc33660c371 00007fc336608edf 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc
0113 08:59:02.922178 (+468614us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.922426 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46577408}
I20250113 08:59:02.924981 12724 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:03.076017 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 512) took 2753 ms. Trace:
I20250113 08:59:03.076205 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:03.007905 14166 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:03.074894 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 282883us
I20250113 08:59:03.006278 14265 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:03.083276 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:03.008533 14246 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:02.959023 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2139) took 1377 ms. Trace:
I20250113 08:59:03.055508 14311 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:03.085489 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:03.092588 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:03.090559 12006 leader_election.cc:336] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.121s (SENT)
I20250113 08:59:03.072154 14322 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:03.113766 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:03.114459 12034 raft_consensus.cc:2388] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:03.075613 14182 consensus_queue.cc:1046] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
W20250113 08:59:03.052728 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 45231.3us
I20250113 08:59:03.116852 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:03.053042 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2128) took 1687 ms (client timeout 1844 ms). Trace:
I20250113 08:59:03.117578 12033 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:03.118474 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:02.738242 12718 rpcz_store.cc:276] 0113 08:58:58.759181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:59.148982 (+389801us) service_pool.cc:225] Handling call
0113 08:58:59.182577 (+ 33595us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:01.562358 (+2379781us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.554162 (+991804us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b54006bdad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:02.554426 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5467129728}
I20250113 08:59:03.125174 12040 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:03.136620 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:03.137409 12037 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:03.137295 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:03.084425 12036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1586) took 1449 ms (client timeout 1774 ms). Trace:
I20250113 08:59:03.119709 14141 raft_consensus.cc:695] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:03.066346 12717 rpcz_store.cc:269] 0113 08:59:01.570297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.590380 (+ 20083us) service_pool.cc:225] Handling call
0113 08:59:02.721289 (+1130909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.094281 14302 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.170054 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:03.105968 14180 raft_consensus.cc:695] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:03.218446 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 137775us
I20250113 08:59:03.169536 14121 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.099730 14210 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:59:03.496603 14166 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:03.485733 11858 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:03.482072 14152 consensus_meta.cc:287] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.770s user 0.008s sys 0.001s
W20250113 08:59:03.541927 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 56254.1us
I20250113 08:59:03.487061 11859 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:03.138813 14201 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.487704 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:03.550418 11885 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:03.489953 12726 rpcz_store.cc:276] 0113 08:59:00.322513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:00.322651 (+ 138us) service_pool.cc:225] Handling call
0113 08:59:00.330806 (+ 8155us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:00.347859 (+ 17053us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:00.347899 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:00.352487 (+ 4588us) log.cc:844] Serialized 26 byte log entry
0113 08:59:00.992706 (+640219us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:00.992753 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:00.992828 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:02.425486 (+1432658us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:03.068720 (+643234us) raft_consensus.cc:1680] finished
0113 08:59:03.068742 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:03.075943 (+ 7201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9550,"raft.queue_time_us":9829,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":4480,"spinlock_wait_cycles":18347776,"thread_start_us":466848,"threads_started":1,"wal-append.queue_time_us":773449}
I20250113 08:59:03.487902 12723 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:03.495429 14265 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.487668 12038 rpcz_store.cc:276] 0113 08:59:01.581576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.159289 (+577713us) service_pool.cc:225] Handling call
0113 08:59:02.958946 (+799657us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.501575 14273 raft_consensus.cc:2743] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:03.503649 14262 consensus_queue.cc:1046] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:03.556653 12726 raft_consensus.cc:1235] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:03.556910 11934 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:03.525354 14013 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:03.216148 11889 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1699) took 1740 ms (client timeout 1659 ms). Trace:
I20250113 08:59:03.502458 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:03.500320 14121 raft_consensus.cc:3049] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
W20250113 08:59:03.139617 14232 consensus_meta.cc:287] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.950s user 0.011s sys 0.000s
I20250113 08:59:03.487020 12041 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 2.
W20250113 08:59:03.585417 11889 rpcz_store.cc:269] 0113 08:59:01.475912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.481317 (+ 5405us) service_pool.cc:225] Handling call
0113 08:59:03.215950 (+1734633us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b540026d254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:03.216074 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3402589440}
W20250113 08:59:03.586346 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 473) took 1822 ms (client timeout 2243 ms). Trace:
I20250113 08:59:03.586624 12724 raft_consensus.cc:2388] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:59:03.586881 12041 rpcz_store.cc:269] 0113 08:59:01.764369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.169756 (+1405387us) service_pool.cc:225] Handling call
0113 08:59:03.586272 (+416516us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.588053 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.588214 12684 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:03.192193 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 71017us
I20250113 08:59:03.143177 11931 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:03.486529 11960 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:03.589735 12723 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:59:03.590122 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:03.485766 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2127) took 2134 ms (client timeout 1545 ms). Trace:
I20250113 08:59:03.497630 14129 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.089749 14325 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:03.337065 14220 consensus_meta.cc:287] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.593s user 0.005s sys 0.004s
W20250113 08:59:03.490682 12042 rpcz_store.cc:269] 0113 08:59:01.365649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.365870 (+ 221us) service_pool.cc:225] Handling call
0113 08:59:02.431855 (+1065985us) spinlock_profiling.cc:243] Waited 7.37 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:03.052941 (+621086us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16959872}
I20250113 08:59:03.209481 14239 consensus_queue.cc:1046] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.352s
I20250113 08:59:03.112111 14015 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:03.139979 14206 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:02.782996 12720 rpcz_store.cc:276] 0113 08:59:01.463508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.740129 (+1276621us) service_pool.cc:225] Handling call
0113 08:59:02.740474 (+ 345us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:02.741028 (+ 554us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:02.741058 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:02.741118 (+ 60us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:02.741180 (+ 62us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:02.741274 (+ 94us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:02.741403 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89}
I20250113 08:59:03.210551 12686 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:03.141673 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:03.613026 11967 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:03.613289 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 402796us
I20250113 08:59:03.193301 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:03.613726 12724 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:03.614102 11964 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:03.614604 12686 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:03.614944 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:03.489521 11965 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:03.218006 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:03.238860 14333 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.414s
I20250113 08:59:03.487762 11884 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:03.487428 11891 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:03.583709 14121 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:03.491220 12006 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:03.201593 14252 consensus_queue.cc:1159] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" responder_term: 5 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.6-4.6]" } } last_received_current_leader { term: 0 index: 0 } }
I20250113 08:59:03.500451 14188 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:03.499826 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:03.488523 14288 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.422s user 0.046s sys 0.104s
I20250113 08:59:03.497799 14246 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:03.504678 14141 consensus_queue.cc:237] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:03.343226 14267 consensus_meta.cc:287] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.192s user 0.006s sys 0.007s
I20250113 08:59:03.138424 14291 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:03.646286 11966 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:03.617837 14323 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:59:03.311439 14315 consensus_meta.cc:287] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.579s user 0.007s sys 0.002s
W20250113 08:59:03.211210 14324 log.cc:926] Time spent T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20250113 08:59:03.540747 12033 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:03.496959 14180 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:03.544436 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 57465.7us
I20250113 08:59:03.351922 12721 raft_consensus.cc:1235] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:03.331925 14321 log.cc:926] Time spent T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20250113 08:59:03.214473 14169 raft_consensus.cc:3049] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:59:03.336400 14112 raft_consensus.cc:3049] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Stepping down as leader of term 2
W20250113 08:59:03.484350 12034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1588) took 1818 ms (client timeout 2090 ms). Trace:
I20250113 08:59:03.553453 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:03.664369 12034 rpcz_store.cc:269] 0113 08:59:01.666431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.108964 (+1442533us) service_pool.cc:225] Handling call
0113 08:59:03.484283 (+375319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.665465 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:03.232547 14320 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:03.669552 14320 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.670356 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2105) took 1239 ms. Trace:
I20250113 08:59:03.339440 14032 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:03.557873 12035 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:03.644042 12717 raft_consensus.cc:2371] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5. Re-sending same reply.
I20250113 08:59:03.507658 14326 consensus_queue.cc:1046] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:03.348137 14303 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.465577 14296 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.199838 14334 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:03.576153 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 760) took 2069 ms. Trace:
I20250113 08:59:03.687790 12726 rpcz_store.cc:276] 0113 08:59:01.488746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.551977 (+2063231us) service_pool.cc:225] Handling call
0113 08:59:03.556403 (+ 4426us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:03.558046 (+ 1643us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:03.558253 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.563913 11962 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:03.584875 11885 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:03.664172 14112 raft_consensus.cc:738] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:03.590869 12040 rpcz_store.cc:269] 0113 08:59:01.350963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.351086 (+ 123us) service_pool.cc:225] Handling call
0113 08:59:02.671176 (+1320090us) spinlock_profiling.cc:243] Waited 6.65 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:03.485652 (+814476us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15297280}
I20250113 08:59:03.662034 14169 raft_consensus.cc:738] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:03.590735 11960 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:59:03.656173 14342 raft_consensus.cc:2743] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:59:03.599934 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:03.106252 14211 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:03.540473 14271 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:03.588797 12041 raft_consensus.cc:2371] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5. Re-sending same reply.
I20250113 08:59:03.653839 14267 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.698009 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:03.698316 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:03.488974 12036 rpcz_store.cc:269] 0113 08:59:01.634752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.151435 (+516683us) service_pool.cc:225] Handling call
0113 08:59:03.076509 (+925074us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.558038 14213 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:03.609356 12007 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:03.700481 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 91237.4us
I20250113 08:59:03.701448 12007 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:03.701452 14032 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:03.701841 14265 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:03.702286 12007 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:03.609584 12722 raft_consensus.cc:604] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:03.563231 14305 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:03.488443 11963 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:03.614571 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 762) took 2069 ms. Trace:
I20250113 08:59:03.620061 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:03.557950 14150 consensus_queue.cc:1035] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20250113 08:59:03.625820 11932 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:03.557765 14290 consensus_queue.cc:1035] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20250113 08:59:03.629032 11893 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:59:03.605293 14182 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:03.201452 14194 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.626077 12685 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:03.607158 14166 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:03.636183 14263 consensus_meta.cc:287] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.503s user 0.009s sys 0.000s
I20250113 08:59:03.713753 14182 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:03.714360 14182 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:03.711434 14210 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:03.715555 14210 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:03.715957 14210 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:03.488458 13989 ts_tablet_manager.cc:1462] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Trace:
0113 08:58:46.832963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:46.833122 (+ 159us) service_pool.cc:225] Handling call
0113 08:58:47.063362 (+230240us) ts_tablet_manager.cc:928] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Initiating tablet copy from peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
0113 08:58:50.854365 (+3791003us) tablet_copy_client.cc:368] Tablet copy session begun
0113 08:58:53.714875 (+2860510us) tablet_metadata.cc:641] Metadata flushed
0113 08:58:53.734814 (+ 19939us) tablet_copy_client.cc:497] Wrote new tablet metadata
0113 08:58:54.297969 (+563155us) tablet_copy_client.cc:418] Wrote new consensus metadata
0113 08:58:54.298292 (+ 323us) tablet_replica.cc:206] Creating consensus instance
0113 08:58:54.298617 (+ 325us) ts_tablet_manager.cc:983] Replica registered.
0113 08:58:54.298677 (+ 60us) inbound_call.cc:173] Queueing success response
0113 08:58:56.669156 (+2370479us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0113 08:59:01.445238 (+4776082us) ts_tablet_manager.cc:1429] Starting tablet replica
0113 08:59:01.446081 (+ 843us) tablet_replica.cc:260] Starting instrumentation
0113 08:59:01.446863 (+ 782us) tablet_replica.cc:279] Starting consensus
Metrics: {"dns_us":41598,"raft.queue_time_us":499660,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"tablet-copy.queue_time_us":131385,"tablet-download-pool.queue_time_us":401,"tablet-download-pool.run_cpu_time_us":1918,"tablet-download-pool.run_wall_time_us":1838214,"thread_start_us":167636,"threads_started":2}
I20250113 08:59:03.694418 14152 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader election won for term 2
W20250113 08:59:03.717633 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 91630.1us
I20250113 08:59:03.719022 12685 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:03.718497 14112 consensus_queue.cc:260] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.719537 14194 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:03.643738 11965 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:03.533645 14322 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.539990 11856 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:03.692406 14323 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:03.694664 14211 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:03.725471 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 185568us
I20250113 08:59:03.726799 11856 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:03.481626 14216 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:03.647468 11967 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:03.656390 12033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2135) took 2155 ms (client timeout 2205 ms). Trace:
W20250113 08:59:03.656384 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 565813us
I20250113 08:59:03.248406 14308 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.630007 14242 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:03.604852 14232 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.673985 14121 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.607764 14220 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.639748 14344 consensus_queue.cc:1046] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:03.668910 14246 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.670883 11968 rpcz_store.cc:276] 0113 08:59:02.431107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.431412 (+ 305us) service_pool.cc:225] Handling call
0113 08:59:02.431770 (+ 358us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:02.432166 (+ 396us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:02.432194 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:02.432644 (+ 450us) log.cc:844] Serialized 26 byte log entry
0113 08:59:03.107501 (+674857us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:03.107540 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:03.107608 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:03.670067 (+562459us) raft_consensus.cc:1680] finished
0113 08:59:03.670096 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:03.670272 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":272909,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":668497,"threads_started":1,"wal-append.queue_time_us":652404}
W20250113 08:59:03.676270 12035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1599) took 1934 ms (client timeout 2166 ms). Trace:
W20250113 08:59:03.558317 11892 consensus_meta.cc:287] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.816s user 0.012s sys 0.003s
I20250113 08:59:03.677371 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:03.655827 14315 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:03.660643 11959 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:03.659040 14291 raft_consensus.cc:695] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:03.656122 14314 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.526729 14301 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:03.590905 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 447803us
I20250113 08:59:03.590279 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 761) took 2061 ms. Trace:
I20250113 08:59:03.689807 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:03.691315 11934 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:03.694738 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.558151 14165 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:03.694429 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.680308 14318 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:03.699117 11960 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:59:03.560688 14212 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.109s user 0.010s sys 0.008s
I20250113 08:59:03.699347 11859 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:03.702064 14032 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:03.616219 12686 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:03.704981 12724 rpcz_store.cc:276] 0113 08:59:01.544858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.587241 (+2042383us) service_pool.cc:225] Handling call
0113 08:59:03.613484 (+ 26243us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:03.614259 (+ 775us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:03.614499 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.611485 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:03.928817 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 312630us
I20250113 08:59:03.688288 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 758) took 2218 ms. Trace:
I20250113 08:59:03.605985 14311 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:03.612179 14015 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:03.717015 11963 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:03.720234 14277 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:03.945936 11963 raft_consensus.cc:1235] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:03.720526 12038 raft_consensus.cc:2371] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3. Re-sending same reply.
I20250113 08:59:03.719866 14112 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:03.720738 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:03.947081 11885 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:03.702603 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.629406 14193 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:03.728008 11965 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:03.883976 11931 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:03.884569 12041 raft_consensus.cc:2371] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3. Re-sending same reply.
I20250113 08:59:03.967773 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2158) took 1181 ms. Trace:
I20250113 08:59:03.968207 12041 rpcz_store.cc:276] 0113 08:59:02.786435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.694496 (+908061us) service_pool.cc:225] Handling call
0113 08:59:03.967693 (+273197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.884655 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:56200 (request call id 0) took 2423 ms. Trace:
I20250113 08:59:03.969280 12722 rpcz_store.cc:276] 0113 08:59:01.467853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.567356 (+ 99503us) service_pool.cc:225] Handling call
0113 08:59:03.884473 (+2317117us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54006004d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336603bc3 00007fc33b56a80a 00007fc334c43d14 00007fc334c43ca2 00007fc334c43bd1 00007fc334c43b4c 00007fc334c42e02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:03.884552 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4387515648}
I20250113 08:59:03.970927 12722 raft_consensus.cc:1235] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:03.971724 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 668) took 2251 ms. Trace:
I20250113 08:59:03.972095 12722 rpcz_store.cc:276] 0113 08:59:01.720379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.970361 (+2249982us) service_pool.cc:225] Handling call
0113 08:59:03.970683 (+ 322us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:03.971453 (+ 770us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:03.971660 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.885506 12723 rpcz_store.cc:276] 0113 08:59:01.528456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.552518 (+2024062us) service_pool.cc:225] Handling call
0113 08:59:03.589426 (+ 36908us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:03.590069 (+ 643us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:03.590231 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.885139 12040 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:03.973479 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2156) took 1205 ms. Trace:
I20250113 08:59:03.973784 12040 rpcz_store.cc:276] 0113 08:59:02.768512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.689539 (+921027us) service_pool.cc:225] Handling call
0113 08:59:03.973428 (+283889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.974845 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:03.886870 12006 leader_election.cc:336] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.161s (SENT)
I20250113 08:59:03.975706 12006 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:03.882567 12037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2134) took 2402 ms (client timeout 2091 ms). Trace:
W20250113 08:59:03.887253 12035 rpcz_store.cc:269] 0113 08:59:01.741699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.116611 (+1374912us) service_pool.cc:225] Handling call
0113 08:59:03.676211 (+559600us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.977982 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:03.978794 12035 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:03.979902 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:03.980614 12035 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:03.888304 12042 raft_consensus.cc:2371] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3. Re-sending same reply.
I20250113 08:59:03.981524 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2157) took 1210 ms. Trace:
I20250113 08:59:03.981931 12042 rpcz_store.cc:276] 0113 08:59:02.770818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.694159 (+923341us) service_pool.cc:225] Handling call
0113 08:59:03.981471 (+287312us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.709205 14305 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:03.899659 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 273892us
W20250113 08:59:03.886763 14338 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.226s user 0.000s sys 0.001s
W20250113 08:59:03.777051 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 50293us
I20250113 08:59:03.996542 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:03.803567 11962 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:03.729585 12007 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:03.998227 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 268744us
I20250113 08:59:03.713857 14263 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.700174 14213 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:03.894106 12033 rpcz_store.cc:269] 0113 08:59:01.500866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.102112 (+1601246us) service_pool.cc:225] Handling call
0113 08:59:03.656329 (+554217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.924288 11858 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:03.889691 11892 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:59:03.927660 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 228390us
I20250113 08:59:03.776700 14201 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:03.931624 12686 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:03.933573 11967 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250113 08:59:03.925768 14196 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:03.923744 14165 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:03.934897 14032 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:03.935883 14344 raft_consensus.cc:3049] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:03.936664 12724 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:03.937722 14220 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:03.938779 12034 raft_consensus.cc:2388] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:03.938808 11891 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:03.853178 14340 log.cc:926] Time spent T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20250113 08:59:03.767673 14245 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:03.962918 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 271714us
I20250113 08:59:03.798493 14345 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:04.029551 14165 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.042351 14220 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.043058 12040 raft_consensus.cc:2388] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:04.043623 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2160) took 1168 ms. Trace:
I20250113 08:59:04.042989 14220 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.043980 12040 rpcz_store.cc:276] 0113 08:59:02.875057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.974526 (+1099469us) service_pool.cc:225] Handling call
0113 08:59:04.043560 (+ 69034us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.035835 14196 raft_consensus.cc:695] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:03.934454 14318 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.036700 14265 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20250113 08:59:04.045971 14196 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.047462 14318 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.050809 14015 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:04.012365 12036 raft_consensus.cc:2371] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 2. Re-sending same reply.
I20250113 08:59:04.052044 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2159) took 1261 ms. Trace:
I20250113 08:59:04.052433 12036 rpcz_store.cc:276] 0113 08:59:02.791098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.702227 (+911129us) service_pool.cc:225] Handling call
0113 08:59:04.051975 (+349748us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.053840 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.054309 14220 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.054630 12036 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:04.055195 14220 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:03.976940 12037 rpcz_store.cc:269] 0113 08:59:01.479990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.771722 (+1291732us) service_pool.cc:225] Handling call
0113 08:59:03.882319 (+1110597us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54000cd5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:03.882468 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":377746304}
I20250113 08:59:04.009074 14342 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:04.056532 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:04.008811 14213 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:04.002259 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 112276us
I20250113 08:59:03.798550 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:04.003396 11859 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:03.706063 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:04.002264 11892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1862) took 2311 ms (client timeout 2177 ms). Trace:
W20250113 08:59:04.061631 11892 rpcz_store.cc:269] 0113 08:59:01.690659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.129534 (+438875us) service_pool.cc:225] Handling call
0113 08:59:04.002190 (+1872656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.064463 12717 raft_consensus.cc:1235] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 3. Ops: []
I20250113 08:59:04.005345 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.981115 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2163) took 1044 ms. Trace:
I20250113 08:59:03.934201 14152 raft_consensus.cc:695] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:03.938508 14121 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.013217 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.946085 14182 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:03.938727 14333 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:04.019979 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:04.103650 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 100369us
I20250113 08:59:03.942391 14242 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:03.880257 14322 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:03.950206 14325 raft_consensus.cc:695] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:04.031122 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 523) took 2348 ms. Trace:
I20250113 08:59:04.119333 12724 rpcz_store.cc:276] 0113 08:59:01.682254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.936083 (+2253829us) service_pool.cc:225] Handling call
0113 08:59:03.936475 (+ 392us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.030801 (+ 94326us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.031047 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:03.962939 14277 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:04.036669 12034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 477) took 1291 ms (client timeout 1703 ms). Trace:
I20250113 08:59:03.995590 14273 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 08:59:04.034386 14166 consensus_queue.cc:1046] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250113 08:59:04.004127 14305 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.018627 14290 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:04.008706 14314 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:04.002061 14124 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.011334 14013 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:04.103152 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 526) took 1184 ms. Trace:
W20250113 08:59:03.926869 14212 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.475s user 0.011s sys 0.009s
I20250113 08:59:04.044981 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:03.941131 12721 rpcz_store.cc:276] 0113 08:59:01.466114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.126258 (+1660144us) service_pool.cc:225] Handling call
0113 08:59:03.218975 (+ 92717us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:03.665159 (+446184us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:03.684785 (+ 19626us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.059783 12723 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:04.164726 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:04.085726 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 198879us
W20250113 08:59:04.015069 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 131190us
I20250113 08:59:04.087977 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:04.066025 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:04.089286 12035 rpcz_store.cc:276] 0113 08:59:02.936587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.979607 (+1043020us) service_pool.cc:225] Handling call
0113 08:59:03.981072 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.090101 12041 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:04.091365 11885 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:04.029925 14344 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:04.087172 12042 raft_consensus.cc:2371] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3. Re-sending same reply.
I20250113 08:59:04.177594 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2161) took 1265 ms. Trace:
W20250113 08:59:04.177819 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 89950.4us
I20250113 08:59:04.177922 12042 rpcz_store.cc:276] 0113 08:59:02.912133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.982675 (+1070542us) service_pool.cc:225] Handling call
0113 08:59:04.177538 (+194863us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.038456 14245 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:04.040271 14201 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:04.026598 14348 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:04.058152 14015 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.110900 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:04.104135 12039 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:59:04.113440 14347 log.cc:926] Time spent T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20250113 08:59:04.113864 14342 raft_consensus.cc:695] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:03.787945 14323 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:04.120347 12034 rpcz_store.cc:269] 0113 08:59:02.745189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.677092 (+931903us) service_pool.cc:225] Handling call
0113 08:59:04.036599 (+359507us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.191187 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:04.121544 12033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2149) took 2536 ms (client timeout 3295 ms). Trace:
W20250113 08:59:04.192296 12033 rpcz_store.cc:269] 0113 08:59:01.584750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.019629 (+2434879us) service_pool.cc:225] Handling call
0113 08:59:04.121466 (+101837us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.056068 14220 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:04.030522 14303 consensus_queue.cc:1046] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20250113 08:59:04.003558 14311 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.158656 12724 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:04.158273 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 521) took 2692 ms. Trace:
I20250113 08:59:04.057192 12037 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:04.195151 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2164) took 1223 ms. Trace:
I20250113 08:59:04.195500 12037 rpcz_store.cc:276] 0113 08:59:02.972004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.056288 (+1084284us) service_pool.cc:225] Handling call
0113 08:59:04.195096 (+138808us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.195693 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.196478 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.110435 14267 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.162763 12040 raft_consensus.cc:2371] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4. Re-sending same reply.
I20250113 08:59:03.776940 14169 consensus_queue.cc:260] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.139776 14216 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:04.137977 14277 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:04.167411 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.200207 11892 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
W20250113 08:59:04.170663 12006 leader_election.cc:336] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.177s (SENT)
I20250113 08:59:04.200881 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 527) took 1121 ms. Trace:
W20250113 08:59:04.201534 12006 leader_election.cc:336] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.191s (SENT)
I20250113 08:59:04.202056 12006 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:04.201311 12724 rpcz_store.cc:276] 0113 08:59:03.079573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.120045 (+1040472us) service_pool.cc:225] Handling call
0113 08:59:04.158424 (+ 38379us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.200630 (+ 42206us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.200823 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.172061 11931 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:04.173337 11961 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:04.174222 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:04.176690 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 765) took 1429 ms. Trace:
I20250113 08:59:04.205085 12723 rpcz_store.cc:276] 0113 08:59:02.746846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.973203 (+1226357us) service_pool.cc:225] Handling call
0113 08:59:04.058568 (+ 85365us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.176424 (+117856us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.176633 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.206988 11885 raft_consensus.cc:1235] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250113 08:59:04.208416 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:04.209735 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.210673 11892 raft_consensus.cc:2405] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:04.178632 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:04.082628 14291 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.176519 14314 raft_consensus.cc:695] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:04.215229 11931 leader_election.cc:400] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
I20250113 08:59:04.155153 14290 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:04.185163 14182 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.185705 14296 raft_consensus.cc:3049] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:04.142479 14333 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:04.160670 14129 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:04.186913 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 467942us
I20250113 08:59:04.197604 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2162) took 1273 ms. Trace:
I20250113 08:59:04.197057 12037 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:04.191797 12034 raft_consensus.cc:2371] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6. Re-sending same reply.
I20250113 08:59:04.196905 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.194943 12725 rpcz_store.cc:276] 0113 08:59:01.465990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.097889 (+1631899us) service_pool.cc:225] Handling call
0113 08:59:03.098314 (+ 425us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:03.115667 (+ 17353us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:03.115700 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:03.135307 (+ 19607us) log.cc:844] Serialized 26 byte log entry
0113 08:59:03.578432 (+443125us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:03.578472 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:03.578541 (+ 69us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:04.151581 (+573040us) raft_consensus.cc:1680] finished
0113 08:59:04.151601 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:04.158189 (+ 6588us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33500,"raft.queue_time_us":8722,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":58011,"thread_start_us":359003,"threads_started":1,"wal-append.queue_time_us":475393}
I20250113 08:59:04.176997 14013 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:04.200177 14232 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.213238 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1606) took 1208 ms. Trace:
I20250113 08:59:04.182865 14245 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.183874 14242 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:04.215546 11931 leader_election.cc:403] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250113 08:59:04.168542 14297 consensus_queue.cc:1046] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20250113 08:59:04.188268 14348 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:04.237905 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.241997 11885 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
W20250113 08:59:04.246035 14335 log.cc:926] Time spent T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.657s user 0.003s sys 0.000s
I20250113 08:59:04.194506 14220 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.162514 14124 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.250502 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.183373 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:04.196362 14152 consensus_queue.cc:237] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.190352 14015 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:04.360667 11893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1866) took 2669 ms (client timeout 2225 ms). Trace:
W20250113 08:59:04.406538 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 191441us
I20250113 08:59:04.407016 12033 raft_consensus.cc:2371] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4. Re-sending same reply.
I20250113 08:59:04.408346 11931 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:04.408557 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.162317 12717 rpcz_store.cc:276] 0113 08:59:02.919017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.063602 (+1144585us) service_pool.cc:225] Handling call
0113 08:59:04.064225 (+ 623us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.064947 (+ 722us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.094139 (+ 29192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:04.236874 14248 consensus_meta.cc:287] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.162s user 0.013s sys 0.004s
I20250113 08:59:04.409816 12035 rpcz_store.cc:276] 0113 08:59:03.004878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.173891 (+1169013us) service_pool.cc:225] Handling call
0113 08:59:04.213162 (+ 39271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.419788 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:04.411063 11889 raft_consensus.cc:2388] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:59:04.353185 14332 log.cc:926] Time spent T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Append to log took a long time: real 0.855s user 0.000s sys 0.002s
I20250113 08:59:04.430227 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.431615 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.432307 12034 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:04.255151 12723 raft_consensus.cc:1235] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:04.265630 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.433226 12033 raft_consensus.cc:2371] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5. Re-sending same reply.
I20250113 08:59:04.251714 12006 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:04.382006 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1692) took 2933 ms. Trace:
I20250113 08:59:04.309085 12041 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:04.287892 14262 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.405777 12040 rpcz_store.cc:276] 0113 08:59:02.924093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.044685 (+1120592us) service_pool.cc:225] Handling call
0113 08:59:04.197547 (+152862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.461885 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.462801 12040 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:59:04.406566 11893 rpcz_store.cc:269] 0113 08:59:01.691705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.751440 (+1059735us) service_pool.cc:225] Handling call
0113 08:59:04.360479 (+1609039us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54003af054. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:04.360601 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1993634560}
I20250113 08:59:04.417204 11931 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:04.257679 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 79127.3us
I20250113 08:59:04.465962 11856 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:04.422333 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:04.239841 14209 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:59:04.423671 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:04.248744 14210 consensus_meta.cc:287] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.532s user 0.010s sys 0.003s
I20250113 08:59:04.237061 14351 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:04.237136 14290 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.380053 12718 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:04.269496 14252 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.249742 14169 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:59:04.315133 14239 consensus_queue.cc:1046] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.170s
I20250113 08:59:04.436194 12685 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:04.434734 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:04.441078 11961 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:04.444047 11887 rpcz_store.cc:276] 0113 08:59:01.448962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.449094 (+ 132us) service_pool.cc:225] Handling call
0113 08:59:01.455415 (+ 6321us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:03.490721 (+2035306us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.381658 (+890937us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54003afa54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:04.381910 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3733550848}
W20250113 08:59:04.445063 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 476) took 2471 ms (client timeout 2700 ms). Trace:
I20250113 08:59:04.376171 14182 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.359642 14188 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.270438 12721 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:04.419559 14194 consensus_meta.cc:287] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.651s user 0.003s sys 0.009s
I20250113 08:59:04.465940 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:04.468930 12725 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:04.237548 14325 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.254235 14273 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.217s
I20250113 08:59:04.421674 14265 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:04.350762 14303 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20250113 08:59:04.279505 14333 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.364745 14323 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:04.544560 14206 consensus_meta.cc:287] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.918s user 0.009s sys 0.017s
I20250113 08:59:04.472863 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.239528 14344 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.473870 14013 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.447336 14232 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.239239 14296 raft_consensus.cc:738] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:04.474481 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:04.595863 12723 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:04.476580 14165 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.475862 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:04.477129 12035 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:04.511583 14242 raft_consensus.cc:695] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:04.511678 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 68351.7us
I20250113 08:59:04.512279 14129 raft_consensus.cc:3049] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:04.654028 14129 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:04.269627 14216 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250113 08:59:04.513149 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 390774us
I20250113 08:59:04.655051 14216 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:04.654554 14242 consensus_queue.cc:237] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.655583 14216 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:04.350024 14318 consensus_queue.cc:1046] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.148s
I20250113 08:59:04.411746 14348 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.658584 12037 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:04.659691 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:04.660527 12037 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:04.660976 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 479) took 1655 ms. Trace:
W20250113 08:59:04.534266 12041 rpcz_store.cc:269] 0113 08:59:01.973358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.004955 (+2031597us) service_pool.cc:225] Handling call
0113 08:59:04.444992 (+440037us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.690279 12041 raft_consensus.cc:1235] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:04.690907 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2136) took 3182 ms. Trace:
I20250113 08:59:04.691200 12041 rpcz_store.cc:276] 0113 08:59:01.508137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.689783 (+3181646us) service_pool.cc:225] Handling call
0113 08:59:04.690117 (+ 334us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.690667 (+ 550us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.690839 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.692404 12041 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:04.692930 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2137) took 3182 ms. Trace:
W20250113 08:59:04.580530 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 143696us
I20250113 08:59:04.560776 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:04.582634 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.266841 14212 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.233s
I20250113 08:59:04.512483 14345 consensus_queue.cc:1046] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20250113 08:59:04.702756 12007 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:04.703471 14345 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:04.584093 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:04.710574 12725 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:04.422735 14248 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.545177 14182 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.586263 12722 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:04.477047 14315 consensus_queue.cc:1046] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20250113 08:59:04.701218 12037 rpcz_store.cc:276] 0113 08:59:03.006048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.659402 (+1653354us) service_pool.cc:225] Handling call
0113 08:59:04.660920 (+ 1518us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.283609 14314 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.532560 14166 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3.
I20250113 08:59:04.475296 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.598804 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 767) took 1484 ms. Trace:
I20250113 08:59:04.598685 14206 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.499508 14262 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.500341 14210 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.637765 14188 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:04.638348 14372 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:04.730726 12037 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250113 08:59:04.639436 14301 consensus_meta.cc:287] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.698s user 0.008s sys 0.012s
I20250113 08:59:04.731415 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2144) took 3148 ms. Trace:
W20250113 08:59:04.646612 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2132) took 3196 ms (client timeout 2166 ms). Trace:
W20250113 08:59:04.598629 12036 consensus_meta.cc:287] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.543s user 0.002s sys 0.020s
I20250113 08:59:04.741695 12042 raft_consensus.cc:2388] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:04.628809 12686 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.018s sys 0.034s
W20250113 08:59:04.744031 12686 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.019s sys 0.034s
W20250113 08:59:04.744782 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 813215us
I20250113 08:59:04.745654 12686 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:04.519877 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 83762.1us
I20250113 08:59:04.519994 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2110) took 1331 ms. Trace:
I20250113 08:59:04.747867 12685 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:04.662344 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:04.724840 14345 raft_consensus.cc:695] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:04.666630 14216 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.532878 14252 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.543201 14354 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:04.360301 14342 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.378926 14349 consensus_queue.cc:1046] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.217s
I20250113 08:59:04.682338 12034 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
W20250113 08:59:04.670903 11959 consensus_meta.cc:287] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.732s user 0.018s sys 0.007s
I20250113 08:59:04.693241 12041 rpcz_store.cc:276] 0113 08:59:01.510420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.691981 (+3181561us) service_pool.cc:225] Handling call
0113 08:59:04.692252 (+ 271us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.692703 (+ 451us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.692872 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.700416 14296 consensus_queue.cc:260] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.704538 12007 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:04.684903 14213 consensus_meta.cc:287] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.598s user 0.000s sys 0.021s
I20250113 08:59:04.253695 14263 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:04.719888 12725 raft_consensus.cc:1235] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
W20250113 08:59:04.708806 14350 log.cc:926] Time spent T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.295s user 0.000s sys 0.001s
I20250113 08:59:04.644806 14232 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:04.716264 12726 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.020s sys 0.024s
I20250113 08:59:04.705999 14351 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:59:04.706465 14358 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:04.718284 14194 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:04.732751 12720 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:04.723312 12721 rpcz_store.cc:276] 0113 08:59:03.114248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.183613 (+1069365us) service_pool.cc:225] Handling call
0113 08:59:04.233472 (+ 49859us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.581898 (+348426us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.598729 (+ 16831us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.733886 14165 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.640282 12723 raft_consensus.cc:1235] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:04.736059 12039 rpcz_store.cc:269] 0113 08:59:01.451496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.997473 (+1545977us) service_pool.cc:225] Handling call
0113 08:59:04.646373 (+1648900us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54003692d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:04.646517 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2057989504}
I20250113 08:59:04.736486 12037 rpcz_store.cc:276] 0113 08:59:01.583287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.719937 (+3136650us) service_pool.cc:225] Handling call
0113 08:59:04.720219 (+ 282us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.731160 (+ 10941us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.731352 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:04.792150 12037 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:04.661692 14344 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:04.751819 12035 raft_consensus.cc:1270] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250113 08:59:04.663753 12040 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:04.753621 12008 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
W20250113 08:59:04.736281 14032 consensus_meta.cc:287] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.697s user 0.008s sys 0.004s
I20250113 08:59:04.764387 12686 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:04.421658 14246 consensus_queue.cc:1046] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20250113 08:59:04.735548 14315 raft_consensus.cc:3049] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
W20250113 08:59:04.769721 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 65303.6us
W20250113 08:59:04.772543 12726 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.634s user 0.020s sys 0.024s
I20250113 08:59:04.770630 12034 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:04.741575 12036 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:04.749747 14372 raft_consensus.cc:695] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:04.770035 14213 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.780310 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:04.781100 14273 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:04.739100 14301 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.825223 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:04.792891 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2145) took 3209 ms. Trace:
I20250113 08:59:04.796780 14201 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:04.797695 14206 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.840485 12723 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:04.750101 14348 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.823757 14212 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:04.823530 14273 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:04.801774 12686 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:04.783027 14296 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:04.858672 14290 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.767836 14262 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.821666 14196 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:04.753818 14303 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20250113 08:59:04.821357 12725 raft_consensus.cc:2371] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4. Re-sending same reply.
W20250113 08:59:04.873947 14355 log.cc:926] Time spent T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.244s user 0.000s sys 0.004s
I20250113 08:59:04.780195 14354 raft_consensus.cc:695] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:04.803400 14112 consensus_meta.cc:287] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.810s user 0.008s sys 0.004s
I20250113 08:59:04.839228 14326 consensus_queue.cc:1046] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.005s
I20250113 08:59:04.854501 14352 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:04.855093 14248 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:04.856135 14121 consensus_meta.cc:287] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.753s user 0.016s sys 0.003s
I20250113 08:59:04.856712 14213 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.840982 14182 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.857357 14232 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.858927 14301 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.859792 14124 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.791007 14194 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.898180 14348 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.802549 14032 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.796653 14315 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:04.872881 14277 consensus_meta.cc:287] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.635s user 0.005s sys 0.006s
W20250113 08:59:04.782788 14211 consensus_meta.cc:287] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.835s user 0.012s sys 0.001s
I20250113 08:59:04.849097 14201 raft_consensus.cc:695] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:04.813067 14358 raft_consensus.cc:695] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:04.870175 14273 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.896740 14320 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:04.902717 14277 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.886833 14210 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:04.904467 14360 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250113 08:59:04.913542 14365 log.cc:926] Time spent T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250113 08:59:04.913695 14277 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:04.852439 14345 consensus_queue.cc:237] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:04.915767 14358 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.914896 14252 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.908763 14341 consensus_queue.cc:1046] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
W20250113 08:59:04.927697 14361 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.088s user 0.000s sys 0.004s
I20250113 08:59:04.926265 14354 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.922113 14315 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.939374 14315 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:04.908078 14372 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:04.980455 11884 raft_consensus.cc:2363] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:04.671459 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:04.982306 11932 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.013s sys 0.004s
W20250113 08:59:04.983062 11932 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.013s sys 0.004s
I20250113 08:59:04.700580 11893 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:04.985209 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:04.987025 11889 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:04.986083 11888 raft_consensus.cc:1235] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:04.719257 11890 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.006s sys 0.052s
I20250113 08:59:04.984992 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:04.747730 11961 rpcz_store.cc:276] 0113 08:59:03.188460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.188611 (+ 151us) service_pool.cc:225] Handling call
0113 08:59:04.065809 (+877198us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b58001c4198. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:04.519930 (+454121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8554368}
W20250113 08:59:04.761636 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 509989us
I20250113 08:59:05.003207 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2122) took 1281 ms. Trace:
I20250113 08:59:05.004657 12006 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:04.872489 12723 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:04.863209 12035 rpcz_store.cc:276] 0113 08:59:01.583578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.751188 (+3167610us) service_pool.cc:225] Handling call
0113 08:59:04.751497 (+ 309us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.792654 (+ 41157us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.792836 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:04.822402 12036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 481) took 1604 ms (client timeout 1868 ms). Trace:
I20250113 08:59:04.804478 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1587) took 3169 ms. Trace:
I20250113 08:59:05.101787 12037 rpcz_store.cc:276] 0113 08:59:01.635058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.791636 (+3156578us) service_pool.cc:225] Handling call
0113 08:59:04.791971 (+ 335us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.804174 (+ 12203us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.804407 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41472}
I20250113 08:59:05.104626 12037 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:04.997519 12039 raft_consensus.cc:1235] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
W20250113 08:59:04.866772 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 65096.3us
I20250113 08:59:05.105279 14137 raft_consensus.cc:3049] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:05.105921 14137 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:05.108121 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2148) took 3523 ms. Trace:
I20250113 08:59:05.108026 12686 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:05.107796 14137 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:05.000506 11890 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s user 0.007s sys 0.052s
I20250113 08:59:04.766280 11959 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:05.004873 12685 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:05.006040 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2123) took 1284 ms. Trace:
I20250113 08:59:04.995816 12033 raft_consensus.cc:2388] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:05.006156 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:05.007534 11967 raft_consensus.cc:2388] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:04.799669 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2146) took 3215 ms. Trace:
I20250113 08:59:04.901165 14032 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:04.902782 14211 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.011015 14123 raft_consensus.cc:3049] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:04.909895 14216 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:04.862600 12721 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEARNER]: Advancing to term 4
I20250113 08:59:05.016666 12725 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250113 08:59:05.021755 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2109) took 1885 ms (client timeout 1908 ms). Trace:
I20250113 08:59:05.003679 11963 rpcz_store.cc:276] 0113 08:59:03.721159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.964207 (+243048us) service_pool.cc:225] Handling call
0113 08:59:03.964567 (+ 360us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:04.030279 (+ 65712us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:04.030313 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:04.039860 (+ 9547us) log.cc:844] Serialized 26 byte log entry
0113 08:59:04.448685 (+408825us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:59:04.448740 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.448831 (+ 91us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:04.871210 (+422379us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b4800240258. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33b4cd581 00007fc33b4c4677 00007fc33b4c450a 00007fc33b5b89bf 00007fc33b5b8687 00007fc33b5ccb8c 00007fc33b5ccb38 00007fc33b5ccab8 00007fc33b5cca73 00007fc33b5cbd1c 00007fc33a538d16
0113 08:59:05.002454 (+131244us) raft_consensus.cc:1680] finished
0113 08:59:05.002492 (+ 38us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:05.002953 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63011,"raft.queue_time_us":67547,"raft.run_cpu_time_us":3631,"raft.run_wall_time_us":841470,"spinlock_wait_cycles":37967616,"thread_start_us":353170,"threads_started":1,"wal-append.queue_time_us":584873}
I20250113 08:59:05.026008 14201 consensus_queue.cc:237] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.084787 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 522) took 3615 ms. Trace:
I20250113 08:59:04.900892 14194 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.108659 12039 rpcz_store.cc:276] 0113 08:59:01.584471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.791628 (+3207157us) service_pool.cc:225] Handling call
0113 08:59:04.791971 (+ 343us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.105554 (+313583us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.107724 (+ 2170us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54000ccbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.108020 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":471118080}
I20250113 08:59:05.111354 11965 rpcz_store.cc:276] 0113 08:59:03.721499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.996629 (+275130us) service_pool.cc:225] Handling call
0113 08:59:03.996949 (+ 320us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:04.021349 (+ 24400us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:04.021380 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:04.037729 (+ 16349us) log.cc:844] Serialized 26 byte log entry
0113 08:59:04.506093 (+468364us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:59:04.506153 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.506248 (+ 95us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:05.005463 (+499215us) raft_consensus.cc:1680] finished
0113 08:59:05.005511 (+ 48us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:05.005788 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23990,"raft.queue_time_us":32576,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":3655,"thread_start_us":423215,"threads_started":1,"wal-append.queue_time_us":699566}
I20250113 08:59:05.116564 11892 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:05.176780 12039 raft_consensus.cc:1235] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 2. Ops: []
I20250113 08:59:05.177838 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1596) took 3457 ms. Trace:
I20250113 08:59:05.178282 14288 raft_consensus.cc:3049] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:05.178861 14288 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:05.117682 14129 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.126286 14360 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:05.181349 14360 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:05.181941 14360 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:05.182209 14129 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:05.115628 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1595) took 3400 ms. Trace:
W20250113 08:59:05.139600 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2153) took 2372 ms (client timeout 1536 ms). Trace:
I20250113 08:59:05.140234 12041 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250113 08:59:05.089802 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 85520.8us
W20250113 08:59:05.169610 11962 rpcz_store.cc:269] 0113 08:59:03.136209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.136354 (+ 145us) service_pool.cc:225] Handling call
0113 08:59:05.021262 (+1884908us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b54002f0b54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.021506 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":172858752}
I20250113 08:59:05.184607 12037 rpcz_store.cc:276] 0113 08:59:01.711427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.103381 (+3391954us) service_pool.cc:225] Handling call
0113 08:59:05.104153 (+ 772us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.105151 (+ 998us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.111525 (+ 6374us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.178196 12039 rpcz_store.cc:276] 0113 08:59:01.719973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.175854 (+3455881us) service_pool.cc:225] Handling call
0113 08:59:05.176397 (+ 543us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.177240 (+ 843us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.177459 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:05.139199 12036 rpcz_store.cc:269] 0113 08:59:03.218468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.053388 (+834920us) service_pool.cc:225] Handling call
0113 08:59:04.810867 (+757479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.172176 12718 rpcz_store.cc:276] 0113 08:59:01.466326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.506759 (+2040433us) service_pool.cc:225] Handling call
0113 08:59:03.507215 (+ 456us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.517599 (+1010384us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.081418 (+563819us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b540068ddd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.081901 (+ 483us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2006845824}
I20250113 08:59:05.188979 12034 rpcz_store.cc:276] 0113 08:59:01.583901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.764661 (+3180760us) service_pool.cc:225] Handling call
0113 08:59:04.765035 (+ 374us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.799212 (+ 34177us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.799597 (+ 385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.191012 12034 raft_consensus.cc:1235] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 1. Current term is 3. Ops: []
I20250113 08:59:05.191684 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1601) took 3428 ms. Trace:
I20250113 08:59:05.192039 12034 rpcz_store.cc:276] 0113 08:59:01.763388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.190103 (+3426715us) service_pool.cc:225] Handling call
0113 08:59:05.190705 (+ 602us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.191419 (+ 714us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.191626 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:05.183243 12038 rpcz_store.cc:269] 0113 08:59:02.766854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.553139 (+786285us) service_pool.cc:225] Handling call
0113 08:59:05.138987 (+1585848us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b540021e654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.139268 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128030976}
I20250113 08:59:05.167966 14123 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:05.201941 12006 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:05.202574 14290 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.202819 12038 raft_consensus.cc:1235] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:05.203676 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 199217us
I20250113 08:59:05.204742 14194 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:05.181500 14288 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:05.217891 14210 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:05.219421 12037 raft_consensus.cc:1235] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:05.220974 12039 raft_consensus.cc:1235] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:05.221472 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 53375.5us
I20250113 08:59:05.231704 11892 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:05.278143 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 478) took 2529 ms. Trace:
I20250113 08:59:05.279932 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1597) took 3555 ms. Trace:
I20250113 08:59:05.282958 12038 rpcz_store.cc:276] 0113 08:59:02.745883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.202001 (+2456118us) service_pool.cc:225] Handling call
0113 08:59:05.202470 (+ 469us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.230321 (+ 27851us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.275286 (+ 44965us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.283253 12037 rpcz_store.cc:276] 0113 08:59:01.722734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.199505 (+3476771us) service_pool.cc:225] Handling call
0113 08:59:05.218908 (+ 19403us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.277488 (+ 58580us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.277688 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.282147 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1598) took 3558 ms. Trace:
I20250113 08:59:05.287042 12039 rpcz_store.cc:276] 0113 08:59:01.723551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.219990 (+3496439us) service_pool.cc:225] Handling call
0113 08:59:05.220556 (+ 566us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.281830 (+ 61274us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.282078 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:05.287734 14245 consensus_meta.cc:287] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.867s user 0.011s sys 0.010s
I20250113 08:59:05.288378 14245 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.275419 14194 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:05.289849 12039 raft_consensus.cc:1235] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 4. Ops: []
I20250113 08:59:05.290850 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:05.276609 14329 log.cc:926] Time spent T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 1.725s user 0.000s sys 0.005s
I20250113 08:59:05.297816 12685 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:05.308746 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1605) took 2319 ms. Trace:
I20250113 08:59:05.309434 12039 rpcz_store.cc:276] 0113 08:59:02.975978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.288031 (+2312053us) service_pool.cc:225] Handling call
0113 08:59:05.288895 (+ 864us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.294744 (+ 5849us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.295013 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.310562 14032 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:05.311558 12717 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250113 08:59:05.312501 12038 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:05.312727 14188 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:05.313269 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 111552us
I20250113 08:59:05.313601 14188 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.314230 14352 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:05.315069 14352 raft_consensus.cc:695] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:05.315572 14283 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:05.313354 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2152) took 2547 ms. Trace:
I20250113 08:59:05.316638 12038 rpcz_store.cc:276] 0113 08:59:02.766269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.307653 (+2541384us) service_pool.cc:225] Handling call
0113 08:59:05.312099 (+ 4446us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.313006 (+ 907us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.313277 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.323336 14161 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.324329 14211 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.324142 14161 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:05.295248 14194 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:05.325567 14360 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.326478 14161 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.327778 14360 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.328266 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 482) took 2077 ms. Trace:
W20250113 08:59:05.328790 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 221081us
I20250113 08:59:05.329630 14344 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.330587 14344 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:05.328729 12038 rpcz_store.cc:276] 0113 08:59:03.251142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.322920 (+2071778us) service_pool.cc:225] Handling call
0113 08:59:05.327480 (+ 4560us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.327828 (+ 348us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:05.327872 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:05.327961 (+ 89us) raft_consensus.cc:1640] Marking committed up to 9
0113 08:59:05.328017 (+ 56us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.328092 (+ 75us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:05.328208 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:05.338294 14305 consensus_meta.cc:287] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.176s user 0.016s sys 0.005s
I20250113 08:59:05.339659 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:05.340866 14220 consensus_meta.cc:287] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.918s user 0.011s sys 0.012s
I20250113 08:59:05.339005 14305 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.342272 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2124) took 1272 ms. Trace:
I20250113 08:59:05.346529 11967 rpcz_store.cc:276] 0113 08:59:04.070338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.070531 (+ 193us) service_pool.cc:225] Handling call
0113 08:59:05.341985 (+1271454us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b54001a13d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.342192 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1886367104}
W20250113 08:59:05.373617 14169 consensus_meta.cc:287] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.841s user 0.007s sys 0.013s
I20250113 08:59:05.395548 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:05.403775 11889 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:05.370556 14188 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:05.407541 14326 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.373353 14283 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:05.287075 14123 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:05.414065 12037 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
W20250113 08:59:05.414812 14296 consensus_meta.cc:287] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.545s user 0.010s sys 0.008s
I20250113 08:59:05.414978 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1604) took 2439 ms. Trace:
I20250113 08:59:05.415400 12037 rpcz_store.cc:276] 0113 08:59:02.975632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.300558 (+2324926us) service_pool.cc:225] Handling call
0113 08:59:05.406258 (+105700us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.414660 (+ 8402us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.414911 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.417670 12039 raft_consensus.cc:1235] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250113 08:59:05.418152 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2170) took 1874 ms. Trace:
I20250113 08:59:05.418525 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 483) took 1933 ms. Trace:
I20250113 08:59:05.418613 12033 rpcz_store.cc:276] 0113 08:59:03.544227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.434465 (+890238us) service_pool.cc:225] Handling call
0113 08:59:05.417880 (+983415us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b540021d254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.418068 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1059211520}
I20250113 08:59:05.419116 12039 rpcz_store.cc:276] 0113 08:59:03.484702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.376266 (+1891564us) service_pool.cc:225] Handling call
0113 08:59:05.417231 (+ 40965us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:05.418221 (+ 990us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.418463 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:05.420388 11892 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:05.422897 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2104) took 2330 ms (client timeout 1952 ms). Trace:
W20250113 08:59:05.423341 11959 rpcz_store.cc:269] 0113 08:59:03.092868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.101079 (+ 8211us) service_pool.cc:225] Handling call
0113 08:59:05.422526 (+2321447us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b54001a0ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.422693 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":81688832}
I20250113 08:59:05.424938 11966 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250113 08:59:05.425981 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 108779us
W20250113 08:59:05.426200 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 49675.9us
I20250113 08:59:05.426932 12006 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:05.427443 14360 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.424961 11968 raft_consensus.cc:1235] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:05.402889 14344 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.445617 11887 raft_consensus.cc:2405] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:05.446056 14283 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.447033 14245 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.428260 14360 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.451455 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1716) took 1233 ms. Trace:
I20250113 08:59:05.451926 11884 rpcz_store.cc:276] 0113 08:59:04.218276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.237548 (+ 19272us) service_pool.cc:225] Handling call
0113 08:59:05.451189 (+1213641us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54000604d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.451353 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1318919424}
I20250113 08:59:05.452411 14360 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.453234 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:05.397840 14220 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.448138 14015 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:05.374740 14352 consensus_queue.cc:237] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.478319 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:05.521799 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.050s user 0.002s sys 0.001s
I20250113 08:59:05.455503 14188 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:59:05.542014 14315 consensus_meta.cc:287] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.602s user 0.004s sys 0.011s
I20250113 08:59:05.519992 14220 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:05.521586 14209 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250113 08:59:05.468593 12006 connection.cc:232] RPC call timeout handler was delayed by -2.478058439999927s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
W20250113 08:59:05.553233 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.086s user 0.003s sys 0.002s
I20250113 08:59:05.475723 14288 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:05.554651 11884 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:05.475330 14123 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:05.554643 11932 leader_election.cc:400] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
W20250113 08:59:05.475473 14359 log.cc:926] Time spent T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.742s user 0.002s sys 0.003s
I20250113 08:59:05.521359 14316 raft_consensus.cc:3049] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:05.576556 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:05.560742 11932 leader_election.cc:403] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:05.576828 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 470) took 1164 ms. Trace:
I20250113 08:59:05.581867 11889 rpcz_store.cc:276] 0113 08:59:04.412789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.423191 (+ 10402us) service_pool.cc:225] Handling call
0113 08:59:05.552705 (+1129514us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5400060ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.576754 (+ 24049us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":731516160}
I20250113 08:59:05.528977 11892 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:05.453233 14194 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.475608 14326 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.582703 14305 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:05.619758 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:05.620465 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:05.592442 14316 raft_consensus.cc:738] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:05.635517 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 171097us
I20250113 08:59:05.575762 14188 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:05.634812 12686 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.703s (SENT) [suppressed 30 similar messages]
I20250113 08:59:05.656733 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:05.657191 12685 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:05.662869 11890 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.002s sys 0.030s
I20250113 08:59:05.667371 11932 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:05.666971 11890 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.556s user 0.002s sys 0.030s
W20250113 08:59:05.686025 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 51257.8us
W20250113 08:59:05.677704 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.046s user 0.026s sys 0.014s
W20250113 08:59:05.677119 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.109s user 0.001s sys 0.045s
W20250113 08:59:05.673629 12006 leader_election.cc:336] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.225s (SENT)
W20250113 08:59:05.666445 14333 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.032s sys 0.041s
W20250113 08:59:05.694583 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.246s user 0.002s sys 0.045s
I20250113 08:59:05.695329 12006 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:05.684824 11892 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:05.668951 14242 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.890s user 0.004s sys 0.046s
I20250113 08:59:05.668447 14344 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:05.671312 14318 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.015s sys 0.046s
I20250113 08:59:05.666926 14315 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 5
I20250113 08:59:05.655997 14283 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:05.688382 11884 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:05.691892 14305 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:05.700158 11932 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:05.683636 14239 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.995s user 0.059s sys 0.095s
W20250113 08:59:05.699712 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.188s user 0.026s sys 0.014s
W20250113 08:59:05.699021 14333 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.060s user 0.033s sys 0.041s
I20250113 08:59:05.713753 14150 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:05.713970 14358 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.036s sys 0.042s
W20250113 08:59:05.714730 14358 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s user 0.036s sys 0.042s
W20250113 08:59:05.713781 14372 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.004s sys 0.013s
W20250113 08:59:05.716038 14372 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s user 0.005s sys 0.013s
W20250113 08:59:05.716385 12684 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.006s sys 0.019s
W20250113 08:59:05.716847 12684 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.048s user 0.007s sys 0.019s
W20250113 08:59:05.717355 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.04922e+06us
W20250113 08:59:05.718384 14219 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.795s user 0.038s sys 0.065s
W20250113 08:59:05.718822 14219 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.840s user 0.038s sys 0.065s
W20250113 08:59:05.721637 14212 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.024s sys 0.044s
W20250113 08:59:05.721506 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 44413.3us
W20250113 08:59:05.722131 14212 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.847s user 0.024s sys 0.045s
W20250113 08:59:05.722487 11858 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.013s user 0.024s sys 0.044s
I20250113 08:59:05.722397 14333 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:05.723052 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.067s user 0.025s sys 0.044s
I20250113 08:59:05.723349 14333 raft_consensus.cc:695] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:05.723997 12726 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.021s sys 0.041s
I20250113 08:59:05.724090 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 529) took 2040 ms. Trace:
W20250113 08:59:05.724452 12726 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.926s user 0.021s sys 0.041s
W20250113 08:59:05.714394 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.2649e+06us
W20250113 08:59:05.702834 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 671) took 3254 ms (client timeout 1647 ms). Trace:
I20250113 08:59:05.722401 14212 raft_consensus.cc:695] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:05.723450 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.06781e+06us
I20250113 08:59:05.733201 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:05.737049 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:05.737957 12718 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:05.721446 14283 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.728446 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:05.740792 14283 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:05.741087 12037 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:05.729928 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2116) took 2180 ms. Trace:
W20250113 08:59:05.742390 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.3252e+06us
I20250113 08:59:05.730993 14129 consensus_queue.cc:1046] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20250113 08:59:05.743723 14129 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:05.744216 14129 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:05.744570 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 453) took 4059 ms. Trace:
I20250113 08:59:05.732100 11884 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:05.746135 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2120) took 2160 ms. Trace:
I20250113 08:59:05.746232 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 469) took 1334 ms. Trace:
I20250113 08:59:05.746627 11966 rpcz_store.cc:276] 0113 08:59:03.585915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.930185 (+344270us) service_pool.cc:225] Handling call
0113 08:59:05.424334 (+1494149us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:05.425467 (+ 1133us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.745888 (+320421us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54001a0ed0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.746069 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3435142016}
I20250113 08:59:05.732728 12036 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:59:05.749174 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.750016 14194 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:05.750826 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.751394 14326 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:05.751679 12036 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:05.752964 14188 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:05.745235 14129 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.755460 14129 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:05.757412 11887 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 474) took 1281 ms (client timeout 1636 ms). Trace:
I20250113 08:59:05.686983 11889 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:59:05.758317 12037 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:05.767947 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 670) took 3542 ms. Trace:
W20250113 08:59:05.757930 11887 rpcz_store.cc:269] 0113 08:59:04.475906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.520282 (+ 44376us) service_pool.cc:225] Handling call
0113 08:59:05.757134 (+1236852us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b540026d254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.757334 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1610008960}
I20250113 08:59:05.767133 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 763) took 4214 ms. Trace:
I20250113 08:59:05.733683 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:05.722584 14239 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.177s user 0.060s sys 0.095s
I20250113 08:59:05.734515 14344 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:05.706090 14242 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.047s user 0.004s sys 0.046s
I20250113 08:59:05.734371 14220 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:05.793291 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:05.794315 12008 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.007s sys 0.020s
I20250113 08:59:05.794929 11893 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
W20250113 08:59:05.795090 12008 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.999s user 0.008s sys 0.020s
I20250113 08:59:05.795346 14311 raft_consensus.cc:3049] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:05.790284 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:05.796300 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1720) took 1530 ms. Trace:
W20250113 08:59:05.796759 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 51850.3us
I20250113 08:59:05.796674 11893 rpcz_store.cc:276] 0113 08:59:04.266245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.987394 (+721149us) service_pool.cc:225] Handling call
0113 08:59:04.988079 (+ 685us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.795391 (+807312us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.796036 (+ 645us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b54000d8754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.796211 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1854355840}
I20250113 08:59:05.799484 11893 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 8. Current term is 9. Ops: []
I20250113 08:59:05.800437 14371 consensus_queue.cc:1046] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:05.722509 11932 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:05.746768 11884 rpcz_store.cc:276] 0113 08:59:04.412143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.710003 (+1297860us) service_pool.cc:225] Handling call
0113 08:59:05.731604 (+ 21601us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.745922 (+ 14318us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.746145 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:05.801789 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 79366.5us
I20250113 08:59:05.801996 14371 raft_consensus.cc:3049] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:05.772995 12722 rpcz_store.cc:276] 0113 08:59:02.225674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.017760 (+1792086us) service_pool.cc:225] Handling call
0113 08:59:04.023793 (+ 6033us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.717227 (+693434us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.767496 (+1050269us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54006cb954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.767752 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1284604672}
I20250113 08:59:05.802448 14371 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:05.740181 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 120798us
W20250113 08:59:05.735088 12007 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.903s user 0.018s sys 0.060s
I20250113 08:59:05.734719 12723 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250113 08:59:05.723374 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.25746e+06us
I20250113 08:59:05.764832 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 773) took 1545 ms. Trace:
I20250113 08:59:05.728021 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.745009 11888 rpcz_store.cc:276] 0113 08:59:01.685505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.955072 (+1269567us) service_pool.cc:225] Handling call
0113 08:59:02.955476 (+ 404us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.988577 (+2033101us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.744307 (+755730us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b540034cbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.744495 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4668208000}
W20250113 08:59:05.742419 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2167) took 2239 ms (client timeout 1969 ms). Trace:
I20250113 08:59:05.743775 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2147) took 4159 ms. Trace:
W20250113 08:59:05.707758 14318 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.049s user 0.016s sys 0.047s
W20250113 08:59:05.734016 14369 log.cc:926] Time spent T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20250113 08:59:05.778076 14342 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.009s sys 0.034s
I20250113 08:59:05.801931 11893 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:05.731810 12039 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250113 08:59:05.711581 14305 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:05.726078 12719 rpcz_store.cc:269] 0113 08:59:02.430720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.720028 (+289308us) service_pool.cc:225] Handling call
0113 08:59:05.640336 (+2920308us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54006f04d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.685486 (+ 45150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4280485760}
I20250113 08:59:05.789654 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:05.790271 14373 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.827541 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:05.795821 14311 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:05.774747 11889 raft_consensus.cc:604] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Received request to transfer leadership to TS 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:05.794265 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1872) took 3890 ms. Trace:
I20250113 08:59:05.725790 14316 consensus_queue.cc:260] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:05.759728 12686 leader_election.cc:336] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.145s (SENT)
I20250113 08:59:05.763023 14302 raft_consensus.cc:3049] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:05.784994 12725 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:05.741736 11960 rpcz_store.cc:276] 0113 08:59:03.548997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.925955 (+376958us) service_pool.cc:225] Handling call
0113 08:59:03.926395 (+ 440us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:03.945411 (+ 19016us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:03.945457 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:03.993160 (+ 47703us) log.cc:844] Serialized 26 byte log entry
0113 08:59:04.413843 (+420683us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:04.413899 (+ 56us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.414000 (+ 101us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:05.291790 (+877790us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:05.729325 (+437535us) raft_consensus.cc:1680] finished
0113 08:59:05.729352 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:05.729668 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28230,"raft.queue_time_us":35742,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":18067,"spinlock_wait_cycles":9889536,"thread_start_us":378959,"threads_started":1,"wal-append.queue_time_us":660071}
I20250113 08:59:05.762518 14351 raft_consensus.cc:3049] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:05.811621 11788 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60016 (request call id 361) took 9018 ms. Trace:
W20250113 08:59:05.782846 14314 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.021s sys 0.033s
I20250113 08:59:05.778026 14358 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:59:05.816911 12041 rpcz_store.cc:276] 0113 08:59:01.584188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.766465 (+3182277us) service_pool.cc:225] Handling call
0113 08:59:05.139300 (+372835us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.183632 (+ 44332us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.743540 (+559908us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b540021e650. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.743684 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":856422656}
W20250113 08:59:05.818800 12042 rpcz_store.cc:269] 0113 08:59:03.503107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.183111 (+680004us) service_pool.cc:225] Handling call
0113 08:59:05.741957 (+1558846us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54002b0554. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.742202 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":750878080}
W20250113 08:59:05.795547 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 999065us
I20250113 08:59:05.825129 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:05.807758 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2116) took 2198 ms. Trace:
I20250113 08:59:05.791898 14344 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:05.839866 11968 rpcz_store.cc:276] 0113 08:59:03.608862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.929751 (+320889us) service_pool.cc:225] Handling call
0113 08:59:03.993384 (+ 63633us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.448779 (+1455395us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.807490 (+358711us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540006dbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.807697 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3243241344}
W20250113 08:59:05.831178 12007 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.033s user 0.019s sys 0.061s
I20250113 08:59:05.761503 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:05.828563 11885 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.035s sys 0.069s
I20250113 08:59:05.836303 11889 raft_consensus.cc:612] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250113 08:59:05.822332 14325 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.014s sys 0.007s
W20250113 08:59:05.843667 11885 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s user 0.035s sys 0.070s
I20250113 08:59:05.808832 14141 raft_consensus.cc:3049] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:05.724547 12724 rpcz_store.cc:276] 0113 08:59:03.683475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.216579 (+533104us) service_pool.cc:225] Handling call
0113 08:59:04.268158 (+ 51579us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:04.268626 (+ 468us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:04.268664 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:04.271177 (+ 2513us) log.cc:844] Serialized 26 byte log entry
0113 08:59:04.729011 (+457834us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:04.729059 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:04.729142 (+ 83us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:05.723667 (+994525us) raft_consensus.cc:1680] finished
0113 08:59:05.723706 (+ 39us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:05.723898 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":951,"raft.queue_time_us":5128,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":7882,"spinlock_wait_cycles":1872128,"thread_start_us":378397,"threads_started":1,"wal-append.queue_time_us":1390814}
I20250113 08:59:05.846485 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:05.838748 12037 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:05.827625 14342 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.017s user 0.009s sys 0.035s
I20250113 08:59:05.803963 14188 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:05.825848 14305 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:05.830379 11890 rpcz_store.cc:276] 0113 08:59:01.903367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:02.960522 (+1057155us) service_pool.cc:225] Handling call
0113 08:59:04.003995 (+1043473us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:04.004424 (+ 429us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:04.004464 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:04.060144 (+ 55680us) log.cc:844] Serialized 26 byte log entry
0113 08:59:05.110011 (+1049867us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:05.673450 (+563439us) log.cc:844] Serialized 14 byte log entry
0113 08:59:05.697080 (+ 23630us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.705091 (+ 8011us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:05.718162 (+ 13071us) raft_consensus.cc:1680] finished
0113 08:59:05.718187 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:05.793818 (+ 75631us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000041d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.794117 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18552,"raft.queue_time_us":27595,"raft.run_cpu_time_us":431,"raft.run_wall_time_us":49542,"spinlock_wait_cycles":2398822272,"thread_start_us":1519502,"threads_started":2,"wal-append.queue_time_us":1564592}
I20250113 08:59:05.724835 14333 consensus_queue.cc:237] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:05.842501 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:05.854161 12039 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:05.794281 14220 raft_consensus.cc:695] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:05.844033 14314 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.023s user 0.022s sys 0.034s
I20250113 08:59:05.734632 14212 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.836969 14302 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:05.839254 14351 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:05.803520 14371 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.888366 14371 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:05.891395 14245 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.851289 12033 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:59:05.850426 14325 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.116s user 0.015s sys 0.007s
I20250113 08:59:05.844420 14360 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:05.892180 14245 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.892871 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:05.856789 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.05386e+06us
W20250113 08:59:05.872082 14273 consensus_meta.cc:287] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.925s user 0.012s sys 0.005s
I20250113 08:59:05.883018 11887 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250113 08:59:05.875785 14345 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.003s sys 0.017s
I20250113 08:59:05.875584 14311 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.875664 11787 catalog_manager.cc:5526] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250113 08:59:05.876474 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 116855us
I20250113 08:59:05.877764 11889 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:05.878264 11884 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20250113 08:59:05.878670 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2117) took 2318 ms (client timeout 2874 ms). Trace:
I20250113 08:59:05.779901 12720 rpcz_store.cc:276] 0113 08:59:01.552060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.626368 (+2074308us) service_pool.cc:225] Handling call
0113 08:59:03.672658 (+ 46290us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.790864 (+1118206us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.766522 (+975658us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54006bc954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.766885 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2419290368}
I20250113 08:59:05.908753 11787 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:05.849488 14141 raft_consensus.cc:738] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:05.909150 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:05.817303 12717 rpcz_store.cc:276] 0113 08:59:04.219308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.428683 (+209375us) service_pool.cc:225] Handling call
0113 08:59:04.561468 (+132785us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:05.312103 (+750635us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.764522 (+452419us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54007a0654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.764773 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1632026880}
I20250113 08:59:05.882416 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.907761 11968 raft_consensus.cc:1235] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:05.883364 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.882985 14302 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.884426 14211 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250113 08:59:05.884665 14351 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.891505 14314 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.869607 11893 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:05.869879 11888 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
W20250113 08:59:05.873996 12008 connection.cc:232] RPC call timeout handler was delayed by -1.9603475899999694s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1647s for a response
W20250113 08:59:05.898286 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 231960us
W20250113 08:59:05.906360 11964 rpcz_store.cc:269] 0113 08:59:03.560236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.705723 (+145487us) service_pool.cc:225] Handling call
0113 08:59:05.846600 (+2140877us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b54001ad1d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:05.878431 (+ 31831us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1641350016}
I20250113 08:59:05.855707 14188 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:05.905830 11934 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:05.914345 12037 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:05.918915 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:05.905736 14345 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.972s user 0.003s sys 0.018s
I20250113 08:59:05.910142 11856 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:05.920719 11887 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:59:05.841447 14373 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:05.910061 14311 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:05.898049 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.923629 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:05.930733 12038 raft_consensus.cc:2363] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:05.930889 14220 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.931025 11787 catalog_manager.cc:5526] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 2, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 2 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:05.932543 11934 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:05.933413 12040 raft_consensus.cc:1235] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 4. Ops: []
I20250113 08:59:05.994156 11966 raft_consensus.cc:2405] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:05.933470 12041 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:05.933804 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.934671 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:05.934971 14392 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:05.935902 11959 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250113 08:59:05.938299 14245 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.008891 14245 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:06.009773 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.010841 12042 raft_consensus.cc:2388] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 2: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:06.011198 14320 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.011313 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.011705 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:05.934684 14169 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:06.012764 14169 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:06.013286 14169 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:06.015522 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.016381 14315 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:06.016717 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:05.940220 14302 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.017892 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.018949 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.019757 12041 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:06.020714 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1590) took 4349 ms. Trace:
I20250113 08:59:05.994712 11787 catalog_manager.cc:5526] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a reported cstate change: term changed from 1 to 3, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 3 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:05.940634 12719 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:06.021081 12040 rpcz_store.cc:276] 0113 08:59:01.671727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.793613 (+3121886us) service_pool.cc:225] Handling call
0113 08:59:04.793938 (+ 325us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:06.019520 (+1225582us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.020413 (+ 893us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400248954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:06.020639 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2598167424}
I20250113 08:59:05.943367 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.022373 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.022607 12037 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:06.023583 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:06.023993 12037 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:06.024188 12040 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:05.947499 14137 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:06.026127 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 93670.1us
I20250113 08:59:06.027554 14194 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.028267 14194 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.032838 14150 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.033473 14150 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.947968 11892 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:05.948729 14349 raft_consensus.cc:3049] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:05.948954 14141 consensus_queue.cc:260] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.950666 12007 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:05.837774 11788 rpcz_store.cc:276] 0113 08:58:56.792899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:58:56.796712 (+ 3813us) service_pool.cc:225] Handling call
0113 08:59:05.811321 (+9014609us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:59:00.044230 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:59:00.113030 (+ 68800us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:59:00.113073 (+ 43us) write_op.cc:435] Acquired schema lock
0113 08:59:00.113100 (+ 27us) tablet.cc:659] Decoding operations
0113 08:59:00.223196 (+110096us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:59:00.248316 (+ 25120us) write_op.cc:641] Partition lock acquired for write op
0113 08:59:00.257807 (+ 9491us) tablet.cc:682] Acquiring locks for 6 operations
0113 08:59:00.453809 (+196002us) tablet.cc:698] Row locks acquired
0113 08:59:00.472006 (+ 18197us) write_op.cc:260] PREPARE: finished
0113 08:59:00.472241 (+ 235us) write_op.cc:270] Start()
0113 08:59:00.472418 (+ 177us) write_op.cc:276] Timestamp: P: 1736758740472181 usec, L: 0
0113 08:59:00.472454 (+ 36us) op_driver.cc:348] REPLICATION: starting
0113 08:59:00.500047 (+ 27593us) log.cc:844] Serialized 2539 byte log entry
0113 08:59:01.825568 (+1325521us) op_driver.cc:464] REPLICATION: finished
0113 08:59:02.354661 (+529093us) write_op.cc:301] APPLY: starting
0113 08:59:02.356010 (+ 1349us) tablet.cc:1364] starting BulkCheckPresence
0113 08:59:02.372870 (+ 16860us) tablet.cc:1367] finished BulkCheckPresence
0113 08:59:02.383886 (+ 11016us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:59:02.416873 (+ 32987us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:59:02.704517 (+287644us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0113 08:59:02.704565 (+ 48us) write_op.cc:312] APPLY: finished
0113 08:59:02.704826 (+ 261us) log.cc:844] Serialized 53 byte log entry
0113 08:59:03.186788 (+481962us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:03.219217 (+ 32429us) write_op.cc:454] Released schema lock
0113 08:59:03.233362 (+ 14145us) write_op.cc:341] FINISH: Updating metrics
0113 08:59:04.352519 (+1119157us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:04.355360 (+ 2841us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800445000. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d1204 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:59:04.355502 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:04.355543 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":623,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":456279,"apply.run_cpu_time_us":94408,"apply.run_wall_time_us":1218047,"num_ops":6,"prepare.queue_time_us":758716,"prepare.run_cpu_time_us":13461,"prepare.run_wall_time_us":918730,"raft.queue_time_us":283334,"raft.run_cpu_time_us":39794,"raft.run_wall_time_us":398193,"replication_time_us":1290493,"spinlock_wait_cycles":96557824,"thread_start_us":1407335,"threads_started":5,"wal-append.queue_time_us":1155499,"wal-append.run_cpu_time_us":40677,"wal-append.run_wall_time_us":3661843}]]}
I20250113 08:59:05.953034 14314 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:05.953778 14318 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:59:05.954474 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 665) took 4287 ms. Trace:
I20250113 08:59:06.025398 14277 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:05.933591 11967 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:05.892899 14360 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:05.991794 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 62792.7us
I20250113 08:59:06.040019 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:05.920478 14188 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:05.910535 14273 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.012948 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.028290 11934 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:05.954669 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 44685us
I20250113 08:59:06.045972 12039 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:06.047288 12008 connection.cc:232] RPC call timeout handler was delayed by -1.0259521319999294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
I20250113 08:59:06.053614 12038 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:59:06.056452 11856 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:06.057461 14150 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.058261 11964 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:06.058162 11856 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:06.038239 14349 raft_consensus.cc:738] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:06.059329 14345 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:06.059908 14349 consensus_queue.cc:260] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.061643 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.061861 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.062353 11890 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:59:06.063424 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.069154 11932 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.068173 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.089852 12042 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:06.093524 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.095299 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.100380 11884 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:06.097054 11893 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:06.050302 14325 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:06.049401 14342 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.100323 11892 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:06.058187 14150 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:06.102645 12037 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:06.061211 14349 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:06.107414 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.040014 14371 consensus_queue.cc:1046] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20250113 08:59:06.093315 14318 raft_consensus.cc:695] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:06.044919 12726 rpcz_store.cc:276] 0113 08:59:01.667189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.703799 (+2036610us) service_pool.cc:225] Handling call
0113 08:59:03.709387 (+ 5588us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:04.086570 (+377183us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:04.798110 (+711540us) log.cc:844] Serialized 14 byte log entry
0113 08:59:05.724934 (+926824us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:05.725008 (+ 74us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:05.725111 (+ 103us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.954119 (+229008us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54007503d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:05.954180 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:05.954388 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1557187,"prepare.run_cpu_time_us":120,"prepare.run_wall_time_us":9478,"raft.queue_time_us":113,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":147,"spinlock_wait_cycles":866888960,"thread_start_us":1585551,"threads_started":2,"wal-append.queue_time_us":1114554}
I20250113 08:59:06.128031 14393 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:06.127893 14318 consensus_queue.cc:237] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.128664 14393 raft_consensus.cc:695] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:06.129779 14393 consensus_queue.cc:237] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.135027 14169 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.138006 14169 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.138113 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.138693 14141 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:06.141809 11886 raft_consensus.cc:1235] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:06.143914 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 458) took 4234 ms. Trace:
I20250113 08:59:06.144322 11886 rpcz_store.cc:276] 0113 08:59:01.909082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:03.056434 (+1147352us) service_pool.cc:225] Handling call
0113 08:59:03.137063 (+ 80629us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:06.142265 (+3005202us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.143212 (+ 947us) spinlock_profiling.cc:243] Waited 2.98 s on lock 0x7b54000a3f54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:06.143701 (+ 489us) spinlock_profiling.cc:243] Waited 67.7 ms on lock 0x7b54000a3f50. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:06.143836 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7012249216}
I20250113 08:59:06.107977 14351 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:06.145390 14288 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250113 08:59:06.146420 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 86776.7us
I20250113 08:59:06.147437 11856 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:06.148247 14391 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.148926 14391 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.150177 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.151388 14391 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:06.152573 11884 raft_consensus.cc:1235] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:06.152861 14194 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.153962 11884 pending_rounds.cc:77] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 5
I20250113 08:59:06.154376 11884 pending_rounds.cc:99] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20250113 08:59:06.155841 14013 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.200s user 0.015s sys 0.029s
W20250113 08:59:06.156455 14013 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.313s user 0.015s sys 0.029s
W20250113 08:59:06.096792 14354 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.007s sys 0.036s
I20250113 08:59:06.157404 11891 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:06.157558 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 129407us
W20250113 08:59:06.157735 14271 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.801s user 0.002s sys 0.022s
W20250113 08:59:06.158221 14271 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.987s user 0.002s sys 0.023s
I20250113 08:59:06.158723 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1713) took 1941 ms. Trace:
I20250113 08:59:06.113931 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:06.159359 11934 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:06.162349 14360 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.163602 14373 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.164085 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:06.164713 14344 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:06.165848 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:06.166574 14344 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:06.166857 12720 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:06.168011 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1721) took 1886 ms. Trace:
I20250113 08:59:06.168632 14314 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.170090 11966 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:06.170179 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:06.120679 11893 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:06.113968 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.172343 11967 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:06.122112 12035 raft_consensus.cc:1235] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:06.124419 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.174018 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2150) took 4010 ms. Trace:
I20250113 08:59:06.167138 14344 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:06.174566 14344 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.174394 12035 rpcz_store.cc:276] 0113 08:59:02.163847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.278170 (+3114323us) service_pool.cc:225] Handling call
0113 08:59:05.371627 (+ 93457us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:06.173192 (+801565us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.173810 (+ 618us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54002bd9d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:06.173966 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1712992128}
I20250113 08:59:06.093051 14161 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:06.176456 14161 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:06.168414 11885 rpcz_store.cc:276] 0113 08:59:04.281626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.990524 (+708898us) service_pool.cc:225] Handling call
0113 08:59:04.991146 (+ 622us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:04.992075 (+ 929us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:04.992136 (+ 61us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:04.992950 (+ 814us) log.cc:844] Serialized 26 byte log entry
0113 08:59:05.844133 (+851183us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:05.844170 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:05.844234 (+ 64us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:06.167796 (+323562us) raft_consensus.cc:1680] finished
0113 08:59:06.167818 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:06.167950 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":27188,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":268,"thread_start_us":850631,"threads_started":1,"wal-append.queue_time_us":1113188}
I20250113 08:59:06.159090 11891 rpcz_store.cc:276] 0113 08:59:04.217151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.219494 (+ 2343us) service_pool.cc:225] Handling call
0113 08:59:04.269755 (+ 50261us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:06.157844 (+1888089us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.158462 (+ 618us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54001b0dd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:06.158654 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4340358400}
I20250113 08:59:06.173790 14383 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:06.177876 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.176865 14161 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.011793 14320 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.179387 14383 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:06.180794 14383 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:06.124524 14316 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:06.181732 12725 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:59:06.181365 14320 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.091596 14277 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.185829 14169 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.186323 14169 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.187220 14277 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:06.188235 14169 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.191697 14342 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:06.157312 14354 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.027s user 0.008s sys 0.036s
I20250113 08:59:06.193351 12722 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:06.193751 14342 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:06.188884 14161 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.197239 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.198632 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.199097 11977 tablet_copy_service.cc:342] P 37bd4bed3ee6423c8dac0cef3bf6882a: Request end of tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-fc84bb347ec2413e9188b7253b0812ab received from {username='slave'} at 127.0.0.1:54924
I20250113 08:59:06.199304 11966 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:06.199710 11977 tablet_copy_service.cc:434] P 37bd4bed3ee6423c8dac0cef3bf6882a: ending tablet copy session 5f9c33777f6845c2848a2a6102a1c2ff-fc84bb347ec2413e9188b7253b0812ab on tablet fc84bb347ec2413e9188b7253b0812ab with peer 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:06.154675 11884 raft_consensus.cc:2883] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:06.201393 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.202543 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:06.203286 11968 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:06.122592 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 198134us
I20250113 08:59:06.170396 14273 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:06.223335 11888 raft_consensus.cc:2405] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:06.225950 11934 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:06.229004 11887 raft_consensus.cc:604] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:06.239974 11887 raft_consensus.cc:612] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250113 08:59:06.231783 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.233511 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.235802 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:06.233624 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 95889.9us
I20250113 08:59:06.236792 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.238489 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:06.229583 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:06.243781 11885 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.214934 14150 raft_consensus.cc:695] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:06.254185 11892 raft_consensus.cc:2388] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:06.201556 14344 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.269513 12722 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:06.269374 14392 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:06.270840 12725 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:06.275604 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 49767.7us
I20250113 08:59:06.212867 14188 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.278199 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:06.277426 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 401988us
I20250113 08:59:06.223488 14391 consensus_queue.cc:1046] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20250113 08:59:06.266459 14161 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.299511 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:06.294375 11934 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:06.286494 14169 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.314430 14169 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.314555 14150 consensus_queue.cc:237] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.315922 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.320086 12034 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:06.316617 11964 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:06.321985 11959 raft_consensus.cc:2388] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:06.324376 12722 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:06.294155 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:06.301522 14315 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:06.328526 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 106921us
I20250113 08:59:06.319818 14345 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:06.311722 14265 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.336982 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.323856 14392 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.328087 14123 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:06.356364 11892 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:06.343633 11964 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:06.301895 14129 consensus_queue.cc:1046] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250113 08:59:06.360558 11893 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:06.381338 11961 raft_consensus.cc:1235] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:06.385447 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250113 08:59:06.388041 14123 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:06.388240 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 672) took 1645 ms. Trace:
I20250113 08:59:06.419229 12007 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:06.375720 14333 consensus_queue.cc:1046] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250113 08:59:06.367043 14320 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20250113 08:59:06.395133 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:06.425733 11968 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.335640 12719 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:06.389909 14341 raft_consensus.cc:3049] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
W20250113 08:59:06.430775 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 283463us
I20250113 08:59:06.340685 14305 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.338616 14315 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.431669 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.440201 11893 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.419704 14123 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.354215 14352 consensus_queue.cc:1046] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20250113 08:59:06.454708 11934 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:06.423976 14169 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.456861 12685 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.459014 11934 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:06.428786 14341 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:06.478531 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 99169.5us
I20250113 08:59:06.486758 12040 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:06.423615 14392 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.480865 11856 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:06.429497 14211 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20250113 08:59:06.458362 14333 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.473958 14344 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.471746 14358 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:06.476384 14288 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:06.471601 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.502806 14352 raft_consensus.cc:3049] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:06.488224 14169 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:59:06.504698 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 49033.1us
I20250113 08:59:06.504866 14409 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.531423 14409 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.532423 14201 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 LEADER]: Stepping down as leader of term 8
I20250113 08:59:06.532866 14201 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:06.515542 14333 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.505329 14407 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:06.534236 14407 raft_consensus.cc:695] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:06.533788 14201 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.535380 14333 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.535461 14201 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:06.419552 12721 rpcz_store.cc:276] 0113 08:59:04.742965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:04.802420 (+ 59455us) service_pool.cc:225] Handling call
0113 08:59:04.848633 (+ 46213us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:05.520395 (+671762us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:05.520448 (+ 53us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:05.521112 (+ 664us) log.cc:844] Serialized 26 byte log entry
0113 08:59:06.106155 (+585043us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:06.133791 (+ 27636us) log.cc:844] Serialized 14 byte log entry
0113 08:59:06.133983 (+ 192us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.134084 (+ 101us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:06.387881 (+253797us) raft_consensus.cc:1680] finished
0113 08:59:06.387908 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:06.388088 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13267,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":23796,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":11694,"spinlock_wait_cycles":16768,"thread_start_us":479831,"threads_started":1,"wal-append.queue_time_us":795327}
I20250113 08:59:06.511229 11893 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:06.513434 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:06.539170 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:06.540143 14341 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.541529 14341 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.513528 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:06.542127 12719 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.519210 14169 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:06.543249 14403 log.cc:926] Time spent T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20250113 08:59:06.522912 11959 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:06.543375 11892 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250113 08:59:06.544297 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:06.544559 14273 consensus_queue.cc:1046] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:06.543438 14169 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:06.545794 14137 raft_consensus.cc:3049] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:06.546208 14137 raft_consensus.cc:738] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:06.517359 14352 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:06.526229 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 52780.5us
I20250113 08:59:06.526386 14360 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.438304 14265 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.530836 14318 raft_consensus.cc:2758] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20250113 08:59:06.533599 14409 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.535163 14407 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.459728 14194 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20250113 08:59:06.548280 11858 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.586265 14161 raft_consensus.cc:3049] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:06.586846 14161 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:06.587785 14161 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.549947 11932 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.484849 14188 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.580525 14273 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.506960 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:59:06.554827 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:06.554383 14137 consensus_queue.cc:260] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.529875 12685 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.606211 12725 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:06.593351 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1602) took 4619 ms. Trace:
I20250113 08:59:06.608057 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.594187 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:06.595966 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 66172.9us
I20250113 08:59:06.612135 14392 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.605728 14358 raft_consensus.cc:695] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:06.614087 14358 consensus_queue.cc:237] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.615294 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.592882 12006 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.591871 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:59:06.608112 12034 rpcz_store.cc:276] 0113 08:59:01.973680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.192911 (+3219231us) service_pool.cc:225] Handling call
0113 08:59:05.218054 (+ 25143us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:06.322296 (+1104242us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.593036 (+270740us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003bd654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:06.593280 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2534074240}
I20250113 08:59:06.601438 11961 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.518056 14288 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:06.606551 11965 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:06.605070 11968 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:06.617132 11967 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:06.602427 11856 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:06.620291 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.625047 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:06.625572 11934 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:06.632340 11890 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:06.590013 14188 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:06.632436 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:06.636440 11888 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:06.632498 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.637087 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.590775 14297 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:06.590271 14413 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.069s user 0.003s sys 0.000s
I20250113 08:59:06.595829 14345 raft_consensus.cc:3049] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:06.675904 11932 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.591841 14305 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.706977 11886 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:06.596704 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:06.602557 14311 consensus_meta.cc:287] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.609s user 0.011s sys 0.000s
W20250113 08:59:06.637802 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 45029.2us
I20250113 08:59:06.595577 14137 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.589210 14161 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.708163 12006 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:06.592594 14316 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20250113 08:59:06.596359 14211 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:06.601290 14265 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:06.589987 14273 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.621533 14288 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:06.675720 14406 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:06.758819 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2127) took 1020 ms. Trace:
I20250113 08:59:06.712002 14188 raft_consensus.cc:695] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:06.676183 14391 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:06.781885 11960 raft_consensus.cc:2405] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:06.700917 14345 raft_consensus.cc:738] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:06.788283 14393 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:06.790035 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 82009.3us
I20250113 08:59:06.740473 14318 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:06.792454 14123 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.602599 14360 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.795917 14315 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:06.850471 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.674021 14352 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.856714 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.785746 14211 raft_consensus.cc:695] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:06.802127 12684 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:06.637365 14333 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:06.715590 14344 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:06.872648 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.839684 14391 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:06.873325 12037 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
W20250113 08:59:06.873469 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 271132us
W20250113 08:59:06.815299 11885 consensus_meta.cc:287] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.554s user 0.012s sys 0.000s
I20250113 08:59:06.711490 14297 raft_consensus.cc:695] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:06.741154 14410 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.851414 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.778255 14325 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:06.813840 14342 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:06.716951 14320 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.081s
I20250113 08:59:06.864266 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.890405 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:06.811120 11962 rpcz_store.cc:276] 0113 08:59:05.738459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.738612 (+ 153us) service_pool.cc:225] Handling call
0113 08:59:05.738949 (+ 337us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:05.739274 (+ 325us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:06.223066 (+483792us) log.cc:844] Serialized 14 byte log entry
0113 08:59:06.740179 (+517113us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:06.758343 (+ 18164us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:06.758424 (+ 81us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.758531 (+ 107us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:06.758734 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":641343,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":102,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":2729,"thread_start_us":956967,"threads_started":2}
I20250113 08:59:06.875393 12040 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:59:06.894317 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:06.875775 11885 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:06.881407 12034 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:06.886076 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:06.841064 14314 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:06.898247 11891 raft_consensus.cc:1235] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:06.849046 14194 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:06.890337 12039 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:06.809546 14406 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.919004 12006 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:06.924355 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.929563 11856 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:06.878170 14316 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.892980 14342 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:06.879184 14344 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:06.931260 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.877789 14318 raft_consensus.cc:695] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:06.879997 14391 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.942262 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:06.951352 11892 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:06.902799 14315 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.852164 14393 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.942353 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:06.949995 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:06.953266 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 103572us
I20250113 08:59:06.893803 12721 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:06.903297 14333 raft_consensus.cc:695] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:06.878897 14409 consensus_queue.cc:1046] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20250113 08:59:06.940608 14407 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:06.963915 12686 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:06.926334 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:06.976855 12686 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:06.977516 14354 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:06.950577 14342 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:06.974257 12006 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:06.950780 14344 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:06.952358 14315 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.892815 14410 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:06.918241 14194 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:07.005496 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 203505us
W20250113 08:59:07.007851 14420 log.cc:926] Time spent T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20250113 08:59:06.997509 14242 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:07.012766 14416 log.cc:926] Time spent T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250113 08:59:07.013391 14267 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.956696 14188 consensus_queue.cc:237] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.959820 14150 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
W20250113 08:59:07.020393 14414 log.cc:926] Time spent T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20250113 08:59:06.943482 14345 consensus_queue.cc:260] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.955400 14373 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:06.955665 14305 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:07.039996 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:07.004560 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:07.049489 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 127834us
I20250113 08:59:07.056429 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:07.052764 14354 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:06.956030 14316 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:06.939662 14314 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:07.061206 12684 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:07.064337 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 87603.7us
I20250113 08:59:06.942451 14297 consensus_queue.cc:237] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:07.045475 14201 consensus_meta.cc:287] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.509s user 0.013s sys 0.000s
I20250113 08:59:07.054589 14265 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:07.101935 11856 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:07.067656 14273 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:07.076186 14392 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:07.136150 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:07.137429 11886 raft_consensus.cc:2388] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 3: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:07.094184 14150 raft_consensus.cc:695] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:07.094977 14360 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:07.102737 14123 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:07.053535 14211 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:07.177798 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 82341.1us
I20250113 08:59:07.082759 14373 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:07.182129 14311 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:07.182300 14418 log.cc:926] Time spent T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20250113 08:59:07.074465 14267 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:07.189363 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 406) took 1268 ms. Trace:
W20250113 08:59:07.183451 14419 log.cc:926] Time spent T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
W20250113 08:59:07.184211 14429 log.cc:926] Time spent T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
W20250113 08:59:07.184504 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 307943us
I20250113 08:59:07.185007 14435 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:07.185420 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:07.185566 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 124485us
I20250113 08:59:07.149204 14271 raft_consensus.cc:2758] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20250113 08:59:07.116758 14333 consensus_queue.cc:237] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:07.154143 14345 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:07.154145 14273 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:59:07.095850 14305 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:07.182505 14427 log.cc:926] Time spent T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20250113 08:59:07.115835 14265 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:07.194640 12725 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9.
I20250113 08:59:07.188019 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:07.098728 14242 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:07.103997 14318 consensus_queue.cc:237] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:07.185817 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:07.342048 12720 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:07.341879 11965 rpcz_store.cc:276] 0113 08:59:05.920673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:05.920803 (+ 130us) service_pool.cc:225] Handling call
0113 08:59:07.189172 (+1268369us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540028f0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:07.189307 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":543498496}
I20250113 08:59:07.247897 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:07.343163 11891 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
I20250113 08:59:07.343587 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:07.344317 11965 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 1.
I20250113 08:59:07.344592 14360 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:07.344964 14360 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:07.345355 14360 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:07.243979 11966 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 3.
W20250113 08:59:07.241102 11961 consensus_meta.cc:287] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.609s user 0.004s sys 0.002s
I20250113 08:59:07.354070 11961 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:07.356856 11932 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:07.390262 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 408) took 1341 ms (client timeout 1562 ms). Trace:
I20250113 08:59:07.390383 12685 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:07.175181 14320 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election won for term 7
W20250113 08:59:07.183491 14426 log.cc:926] Time spent T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
W20250113 08:59:07.391075 11960 rpcz_store.cc:269] 0113 08:59:06.049199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.049665 (+ 466us) service_pool.cc:225] Handling call
0113 08:59:07.351033 (+1301368us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b540028e1d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:07.390158 (+ 39125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1575376256}
W20250113 08:59:07.495714 11887 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.052s sys 0.053s
I20250113 08:59:08.054584 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:59:07.473966 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:07.561856 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:07.510385 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 341769us
I20250113 08:59:07.679368 11931 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:08.099449 12686 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:07.833864 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:07.450793 12007 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:07.190059 14392 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:07.230716 14341 consensus_meta.cc:287] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.689s user 0.010s sys 0.002s
I20250113 08:59:07.777930 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 682) took 1444 ms. Trace:
I20250113 08:59:08.098150 11963 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 2.
I20250113 08:59:08.119923 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 489) took 1505 ms. Trace:
W20250113 08:59:08.096994 11887 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.139s user 0.053s sys 0.073s
W20250113 08:59:08.101085 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 421844us
I20250113 08:59:07.494951 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:07.940212 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 120517us
W20250113 08:59:07.518646 14351 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.979s user 0.007s sys 0.012s
I20250113 08:59:07.475055 11962 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:08.101656 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 711344us
I20250113 08:59:07.941495 11889 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:07.545133 12718 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:07.236471 14393 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:08.125623 12722 rpcz_store.cc:276] 0113 08:59:06.333869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.334048 (+ 179us) service_pool.cc:225] Handling call
0113 08:59:06.524501 (+190453us) spinlock_profiling.cc:243] Waited 9.81 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:59:06.524699 (+ 198us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:06.915448 (+390749us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:06.915497 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:06.941684 (+ 26187us) log.cc:844] Serialized 26 byte log entry
0113 08:59:07.156408 (+214724us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:07.156455 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:07.156541 (+ 86us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:07.558605 (+402064us) raft_consensus.cc:1680] finished
0113 08:59:07.558646 (+ 41us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:07.777838 (+219192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2202,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":113,"spinlock_wait_cycles":22555520,"thread_start_us":557515,"threads_started":2,"wal-append.queue_time_us":591712}
I20250113 08:59:07.213982 14314 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:07.548159 12008 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.003s sys 0.011s
W20250113 08:59:07.233891 14425 log.cc:926] Time spent T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250113 08:59:08.095180 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 431) took 1293 ms. Trace:
W20250113 08:59:07.755342 14342 consensus_meta.cc:287] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.706s user 0.008s sys 0.002s
I20250113 08:59:08.129642 11931 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:08.126968 11890 rpcz_store.cc:276] 0113 08:59:06.619647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.633628 (+ 13981us) service_pool.cc:225] Handling call
0113 08:59:06.699098 (+ 65470us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:06.717900 (+ 18802us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:06.717934 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:06.846858 (+128924us) log.cc:844] Serialized 26 byte log entry
0113 08:59:07.010122 (+163264us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b4800080058. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33b4cd581 00007fc33b4c4677 00007fc33b4c450a 00007fc33b5b89bf 00007fc33b5b8687 00007fc33b5ccb8c 00007fc33b5ccb38 00007fc33b5ccab8 00007fc33b5cca73 00007fc33b5cbd1c 00007fc33a538d16
0113 08:59:07.115058 (+104936us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:07.115555 (+ 497us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:07.115624 (+ 69us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:08.096587 (+980963us) raft_consensus.cc:1680] finished
0113 08:59:08.096606 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:08.119850 (+ 23244us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6255,"raft.queue_time_us":6445,"raft.run_cpu_time_us":3323,"raft.run_wall_time_us":298180,"spinlock_wait_cycles":53129984,"thread_start_us":225118,"threads_started":1,"wal-append.queue_time_us":371911}
W20250113 08:59:08.143553 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 692846us
W20250113 08:59:07.338958 14137 consensus_meta.cc:287] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.597s user 0.007s sys 0.002s
I20250113 08:59:08.131701 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2135) took 1114 ms. Trace:
I20250113 08:59:07.449815 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 680) took 1154 ms. Trace:
I20250113 08:59:08.148722 14242 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:08.150019 14452 log.cc:926] Time spent T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20250113 08:59:08.150058 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 683) took 1780 ms. Trace:
I20250113 08:59:07.218456 14150 consensus_queue.cc:237] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:07.215699 14267 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:08.151549 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.148806 12717 rpcz_store.cc:276] 0113 08:59:06.295364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.311415 (+ 16051us) service_pool.cc:225] Handling call
0113 08:59:06.347082 (+ 35667us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:06.450139 (+103057us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:06.459866 (+ 9727us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:06.492759 (+ 32893us) log.cc:844] Serialized 26 byte log entry
0113 08:59:06.952465 (+459706us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:06.952535 (+ 70us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.952633 (+ 98us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:07.215901 (+263268us) raft_consensus.cc:1680] finished
0113 08:59:07.215939 (+ 38us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:07.411739 (+195800us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33619,"raft.queue_time_us":35647,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":17875,"thread_start_us":416270,"threads_started":1,"wal-append.queue_time_us":661148}
I20250113 08:59:08.150686 12723 rpcz_store.cc:276] 0113 08:59:06.370037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.370294 (+ 257us) service_pool.cc:225] Handling call
0113 08:59:06.428491 (+ 58197us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:06.514254 (+ 85763us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:06.514284 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:06.514771 (+ 487us) log.cc:844] Serialized 26 byte log entry
0113 08:59:06.919207 (+404436us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:06.919241 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.919301 (+ 60us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:08.148773 (+1229472us) raft_consensus.cc:1680] finished
0113 08:59:08.148792 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:08.149642 (+ 850us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b5400534dd0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.149981 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76762,"raft.queue_time_us":80346,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":86284,"spinlock_wait_cycles":64256768,"thread_start_us":403949,"threads_started":1,"wal-append.queue_time_us":544662}
I20250113 08:59:08.152887 14358 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:07.228570 14373 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:07.227185 12719 consensus_meta.cc:287] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.685s user 0.014s sys 0.007s
I20250113 08:59:07.243768 14316 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:07.219650 14169 consensus_meta.cc:287] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.674s user 0.004s sys 0.016s
W20250113 08:59:08.020236 12720 consensus_meta.cc:287] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.677s user 0.011s sys 0.000s
I20250113 08:59:07.833563 14129 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.136052 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:07.569301 14305 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:07.414597 11856 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:08.006623 14360 consensus_meta.cc:287] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.661s user 0.009s sys 0.001s
W20250113 08:59:08.112499 14391 consensus_meta.cc:287] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.133s user 0.009s sys 0.003s
I20250113 08:59:07.249334 14406 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:08.112390 12006 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.062s user 0.007s sys 0.007s
W20250113 08:59:08.185053 12006 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.135s user 0.007s sys 0.007s
W20250113 08:59:08.185362 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.22634e+06us
I20250113 08:59:07.234510 14315 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.137627 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.135438 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 546) took 1921 ms. Trace:
I20250113 08:59:07.833853 14271 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:07.512183 14438 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:08.142431 12008 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.104s user 0.008s sys 0.025s
I20250113 08:59:08.142441 11966 rpcz_store.cc:276] 0113 08:59:06.802242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:07.002461 (+200219us) service_pool.cc:225] Handling call
0113 08:59:08.094987 (+1092526us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002a3d54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.095114 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":379002624}
I20250113 08:59:07.732072 14441 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:07.466883 14194 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:07.755059 14141 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:07.197557 14311 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.122579 14446 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:08.122298 14325 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:07.720000 14297 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:07.718226 14161 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.145284 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.145447 11963 rpcz_store.cc:276] 0113 08:59:07.012179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:07.555994 (+543815us) service_pool.cc:225] Handling call
0113 08:59:08.126948 (+570954us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:08.134778 14351 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.593s user 0.016s sys 0.025s
I20250113 08:59:07.881470 14265 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:07.778545 14211 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.000s sys 0.010s
I20250113 08:59:07.518950 14435 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.131492 14392 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:08.178361 14169 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.194026 14391 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:08.147758 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:08.262521 14325 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:08.150816 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.153434 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:08.153580 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:08.138326 12718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 783) took 1368 ms (client timeout 1691 ms). Trace:
I20250113 08:59:08.133502 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:08.084492 14345 consensus_meta.cc:287] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.672s user 0.011s sys 0.000s
I20250113 08:59:08.269035 14449 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:07.224331 14288 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:07.465931 14421 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:08.227263 14441 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:07.314666 14344 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:08.323400 14441 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.112967 14410 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:08.177695 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 425) took 1661 ms. Trace:
W20250113 08:59:08.188580 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 778) took 1665 ms (client timeout 2086 ms). Trace:
I20250113 08:59:07.411739 14407 consensus_queue.cc:1046] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.759s
I20250113 08:59:08.328893 14344 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:08.191903 12726 rpcz_store.cc:276] 0113 08:59:06.214167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.214320 (+ 153us) service_pool.cc:225] Handling call
0113 08:59:06.227751 (+ 13431us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:06.228048 (+ 297us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:06.228084 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:06.284524 (+ 56440us) log.cc:844] Serialized 26 byte log entry
0113 08:59:06.709501 (+424977us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:06.709551 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:06.709645 (+ 94us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:08.135216 (+1425571us) raft_consensus.cc:1680] finished
0113 08:59:08.135237 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:08.135377 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":385662,"threads_started":1,"wal-append.queue_time_us":576294}
I20250113 08:59:08.192939 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2123) took 1185 ms. Trace:
I20250113 08:59:08.329684 14441 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.193661 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.329954 11965 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:08.330323 14441 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.155004 14341 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:08.331346 14341 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:08.191490 11892 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:08.159693 14450 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:08.331707 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:08.332614 14441 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.332779 12726 raft_consensus.cc:2388] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:08.161235 14137 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.333957 14137 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.334782 14288 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.335551 11968 raft_consensus.cc:2363] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4.
I20250113 08:59:08.336439 14137 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.162016 14314 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:08.337224 14314 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:59:08.325182 12725 rpcz_store.cc:269] 0113 08:59:06.514412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.632152 (+117740us) service_pool.cc:225] Handling call
0113 08:59:08.179756 (+1547604us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b54004addd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.179900 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1022982400}
I20250113 08:59:08.337791 14314 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:08.338418 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:08.339077 12725 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 2.
I20250113 08:59:08.340040 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:08.340791 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 686) took 1716 ms. Trace:
I20250113 08:59:08.269927 14194 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20250113 08:59:08.174876 14393 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.342403 14393 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:08.278160 12718 rpcz_store.cc:269] 0113 08:59:06.770263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:07.185067 (+414804us) service_pool.cc:225] Handling call
0113 08:59:08.138236 (+953169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.344852 14393 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.176519 12720 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:08.258163 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.357919 11962 raft_consensus.cc:2405] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250113 08:59:08.194892 14212 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.198221 14242 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:08.264712 12007 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:08.362567 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 97942.4us
I20250113 08:59:08.199389 14406 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250113 08:59:08.266028 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 913690us
I20250113 08:59:08.268213 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:08.260187 14448 log.cc:926] Time spent T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.275s user 0.001s sys 0.001s
I20250113 08:59:08.176091 14358 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.188159 14129 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.259286 14438 raft_consensus.cc:695] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:08.147271 11931 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:08.149721 14342 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.274363 12722 raft_consensus.cc:2405] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250113 08:59:08.267343 14211 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.049s user 0.003s sys 0.011s
I20250113 08:59:08.192229 14360 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.280797 12038 raft_consensus.cc:2388] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:08.136782 14453 log.cc:926] Time spent T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.299s user 0.000s sys 0.001s
I20250113 08:59:08.260567 14141 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:08.272065 14352 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:08.154165 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 774) took 1814 ms. Trace:
I20250113 08:59:08.294466 14421 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:08.296960 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 424) took 1835 ms (client timeout 1567 ms). Trace:
W20250113 08:59:08.397321 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 250124us
I20250113 08:59:08.301604 11959 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250113 08:59:08.267714 14161 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.302385 14392 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:08.303180 14443 log.cc:926] Time spent T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20250113 08:59:08.269131 14444 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:08.304404 14391 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:08.305460 14311 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.305857 14449 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.315701 14169 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.258909 14271 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:08.316800 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 775) took 1937 ms. Trace:
I20250113 08:59:08.186923 12006 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:08.433236 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 246379us
I20250113 08:59:08.332376 14450 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.264670 14297 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.198165 12684 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:08.175282 12719 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:08.324184 11964 rpcz_store.cc:276] 0113 08:59:06.515938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.516119 (+ 181us) service_pool.cc:225] Handling call
0113 08:59:07.189338 (+673219us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:07.681873 (+492535us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:07.681938 (+ 65us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:07.682562 (+ 624us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:07.682621 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:07.682715 (+ 94us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:08.177499 (+494784us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b540028f0d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.177633 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":121855,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":1458080000,"thread_start_us":492006,"threads_started":1}
W20250113 08:59:08.347920 11965 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2131) took 1337 ms (client timeout 1663 ms). Trace:
I20250113 08:59:08.341120 12725 rpcz_store.cc:276] 0113 08:59:06.624851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.339784 (+1714933us) service_pool.cc:225] Handling call
0113 08:59:08.340722 (+ 938us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.349789 11934 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:08.330035 11960 rpcz_store.cc:276] 0113 08:59:07.007771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.119601 (+1111830us) service_pool.cc:225] Handling call
0113 08:59:08.192881 (+ 73280us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:08.440421 11965 rpcz_store.cc:269] 0113 08:59:07.010778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:07.434085 (+423307us) service_pool.cc:225] Handling call
0113 08:59:08.347702 (+913617us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540015de54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.347839 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":520203648}
I20250113 08:59:08.331719 14341 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:59:08.358126 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 243041us
I20250113 08:59:08.358566 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 555) took 1751 ms. Trace:
I20250113 08:59:08.442888 12726 rpcz_store.cc:276] 0113 08:59:06.602220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.331373 (+1729153us) service_pool.cc:225] Handling call
0113 08:59:08.353996 (+ 22623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.358070 11966 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
W20250113 08:59:08.444065 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2134) took 1432 ms (client timeout 1786 ms). Trace:
W20250113 08:59:08.444564 11966 rpcz_store.cc:269] 0113 08:59:07.011832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.199224 (+1187392us) service_pool.cc:225] Handling call
0113 08:59:08.443988 (+244764us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.349295 14407 raft_consensus.cc:3049] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:08.450964 11889 raft_consensus.cc:1270] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20250113 08:59:08.451196 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 101527us
I20250113 08:59:08.367357 11963 raft_consensus.cc:2388] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:08.452255 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 429) took 1650 ms (client timeout 2180 ms). Trace:
I20250113 08:59:08.452553 11934 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:08.452622 11963 rpcz_store.cc:269] 0113 08:59:06.801510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.267999 (+1466489us) service_pool.cc:225] Handling call
0113 08:59:08.452195 (+184196us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:08.365203 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 178899us
W20250113 08:59:08.367776 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 268414us
I20250113 08:59:08.333138 14441 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:08.376015 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 426) took 1772 ms (client timeout 1703 ms). Trace:
W20250113 08:59:08.456665 11968 rpcz_store.cc:269] 0113 08:59:06.603489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.624635 (+ 21146us) service_pool.cc:225] Handling call
0113 08:59:08.375749 (+1751114us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b540006e5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.375927 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3931589504}
I20250113 08:59:08.361739 14242 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:08.383679 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 557) took 1138 ms. Trace:
I20250113 08:59:08.458467 12722 rpcz_store.cc:276] 0113 08:59:07.245678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.143862 (+898184us) service_pool.cc:225] Handling call
0113 08:59:08.383617 (+239755us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.369817 14393 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:08.398527 11931 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:08.361810 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 556) took 1260 ms. Trace:
I20250113 08:59:08.469575 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.469993 12720 rpcz_store.cc:276] 0113 08:59:07.101422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:07.185508 (+ 84086us) service_pool.cc:225] Handling call
0113 08:59:08.360851 (+1175343us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:08.409794 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 779) took 1895 ms (client timeout 2028 ms). Trace:
W20250113 08:59:08.414461 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2121) took 1489 ms (client timeout 1609 ms). Trace:
W20250113 08:59:08.478361 11962 rpcz_store.cc:269] 0113 08:59:06.925264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.963762 (+ 38498us) service_pool.cc:225] Handling call
0113 08:59:08.414185 (+1450423us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b54002a3854. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.414376 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":909781248}
W20250113 08:59:08.478874 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 80467.9us
I20250113 08:59:08.479661 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.477872 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.480484 11962 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:08.416214 14439 log.cc:926] Time spent T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.150s user 0.000s sys 0.003s
I20250113 08:59:08.383545 12724 rpcz_store.cc:276] 0113 08:59:06.340186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.340356 (+ 170us) service_pool.cc:225] Handling call
0113 08:59:06.371806 (+ 31450us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:06.429931 (+ 58125us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:06.429989 (+ 58us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:06.460816 (+ 30827us) log.cc:844] Serialized 26 byte log entry
0113 08:59:06.939706 (+478890us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:07.186686 (+246980us) log.cc:844] Serialized 14 byte log entry
0113 08:59:07.186961 (+ 275us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:07.187051 (+ 90us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:08.153964 (+966913us) raft_consensus.cc:1680] finished
0113 08:59:08.153985 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:08.154111 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56371,"prepare.queue_time_us":555333,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":51334,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":35794,"thread_start_us":614517,"threads_started":2,"wal-append.queue_time_us":582450}
W20250113 08:59:08.397468 11961 rpcz_store.cc:269] 0113 08:59:06.474761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.480323 (+ 5562us) service_pool.cc:225] Handling call
0113 08:59:08.296684 (+1816361us) spinlock_profiling.cc:243] Waited 63.3 ms on lock 0x7b5400196254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.296849 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145556352}
I20250113 08:59:08.481922 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:08.482362 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.370544 14320 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:08.417083 14265 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:59:08.431344 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1630) took 1570 ms (client timeout 1539 ms). Trace:
W20250113 08:59:08.485664 12038 rpcz_store.cc:269] 0113 08:59:06.860538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.860670 (+ 132us) service_pool.cc:225] Handling call
0113 08:59:08.431031 (+1570361us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54003692d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.431238 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3185279872}
I20250113 08:59:08.436187 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 491) took 1393 ms. Trace:
I20250113 08:59:08.438305 11892 raft_consensus.cc:1235] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:08.347270 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:08.416671 14421 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:08.446712 11967 raft_consensus.cc:1235] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250113 08:59:08.451157 14407 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:08.432929 12717 rpcz_store.cc:276] 0113 08:59:06.379520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.153134 (+1773614us) service_pool.cc:225] Handling call
0113 08:59:08.316733 (+163599us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.451457 11889 pending_rounds.cc:77] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 1
I20250113 08:59:08.508157 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.515501 11889 pending_rounds.cc:99] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250113 08:59:08.515777 11889 raft_consensus.cc:2883] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:08.460695 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:08.462013 14447 log.cc:926] Time spent T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.230s user 0.000s sys 0.002s
I20250113 08:59:08.462769 11888 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:08.463379 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.517585 11968 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:08.463627 11856 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:08.464504 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:08.465436 14150 consensus_queue.cc:1035] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:59:08.520934 11932 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:08.466359 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:08.467173 14455 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
W20250113 08:59:08.467324 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 269283us
I20250113 08:59:08.470723 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2118) took 1688 ms. Trace:
I20250113 08:59:08.525141 11964 rpcz_store.cc:276] 0113 08:59:06.782587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.454131 (+1671544us) service_pool.cc:225] Handling call
0113 08:59:08.470643 (+ 16512us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.471859 11934 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:08.455736 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.477825 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:08.482904 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 550) took 1959 ms. Trace:
I20250113 08:59:08.528648 12726 rpcz_store.cc:276] 0113 08:59:06.523316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.454326 (+1931010us) service_pool.cc:225] Handling call
0113 08:59:08.482833 (+ 28507us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.367336 14212 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:08.423831 14438 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:08.477660 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 113348us
I20250113 08:59:08.531042 11960 raft_consensus.cc:2388] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:08.531113 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.531756 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.487181 11931 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:08.488664 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.532732 11892 raft_consensus.cc:2405] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:08.533142 11966 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
W20250113 08:59:08.503279 14454 log.cc:926] Time spent T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.241s user 0.001s sys 0.001s
W20250113 08:59:08.449760 14273 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.023s sys 0.051s
I20250113 08:59:08.469566 14393 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:08.454406 11885 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
W20250113 08:59:08.455729 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 776) took 2011 ms (client timeout 2141 ms). Trace:
I20250113 08:59:08.455740 14435 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.503834 12724 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:08.516556 14318 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.566591 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:08.486881 11884 rpcz_store.cc:276] 0113 08:59:07.042456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:07.042606 (+ 150us) service_pool.cc:225] Handling call
0113 08:59:07.063403 (+ 20797us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:07.078792 (+ 15389us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:07.078833 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:07.098671 (+ 19838us) log.cc:844] Serialized 52 byte log entry
0113 08:59:07.752453 (+653782us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:07.752491 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:07.752559 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:08.435892 (+683333us) raft_consensus.cc:1680] finished
0113 08:59:08.435918 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:08.436095 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20904,"raft.queue_time_us":11534,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":28783,"thread_start_us":400273,"threads_started":1,"wal-append.queue_time_us":769859}
I20250113 08:59:08.503762 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 490) took 1881 ms. Trace:
I20250113 08:59:08.459270 14333 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:08.496346 12722 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:08.534475 14446 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:08.481071 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2139) took 1332 ms. Trace:
W20250113 08:59:08.554208 14273 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.053s user 0.023s sys 0.051s
I20250113 08:59:08.568470 11887 rpcz_store.cc:276] 0113 08:59:06.622500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.673817 (+ 51317us) service_pool.cc:225] Handling call
0113 08:59:06.718414 (+ 44597us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:06.855051 (+136637us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:06.855095 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:06.881092 (+ 25997us) log.cc:844] Serialized 26 byte log entry
0113 08:59:08.129110 (+1248018us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:08.129151 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:08.129222 (+ 71us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:08.415481 (+286259us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:08.503488 (+ 88007us) raft_consensus.cc:1680] finished
0113 08:59:08.503509 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:08.503701 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51360,"raft.queue_time_us":69660,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":9253,"spinlock_wait_cycles":2799872,"thread_start_us":1170552,"threads_started":1,"wal-append.queue_time_us":1267816}
I20250113 08:59:08.532359 11963 raft_consensus.cc:2388] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:08.533771 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2141) took 1039 ms. Trace:
I20250113 08:59:08.453384 14392 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:08.532215 12007 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:08.536274 12684 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:08.528259 11965 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:08.592667 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 56505.7us
I20250113 08:59:08.539084 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2120) took 1667 ms. Trace:
I20250113 08:59:08.425621 14444 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.545497 14455 raft_consensus.cc:695] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:08.542219 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 55123.9us
W20250113 08:59:08.543237 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 88859.1us
W20250113 08:59:08.526482 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 54727.8us
I20250113 08:59:08.560528 14161 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.477947 14458 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:08.513835 12720 raft_consensus.cc:1235] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:08.503783 12718 raft_consensus.cc:1270] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:08.546518 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.547067 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:08.553699 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 99822.5us
I20250113 08:59:08.554363 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:08.561345 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 780) took 2046 ms. Trace:
I20250113 08:59:08.618966 12724 rpcz_store.cc:276] 0113 08:59:06.515221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.481797 (+1966576us) service_pool.cc:225] Handling call
0113 08:59:08.503641 (+ 21844us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:08.560995 (+ 57354us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:08.561249 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.472618 14242 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:08.563504 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.481389 14345 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20250113 08:59:08.576305 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 684) took 2151 ms. Trace:
I20250113 08:59:08.519057 12725 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:08.554387 14188 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:08.583551 14150 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:59:08.595312 14392 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:08.488426 14169 consensus_queue.cc:1046] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20250113 08:59:08.587539 11962 rpcz_store.cc:276] 0113 08:59:07.148293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.479367 (+1331074us) service_pool.cc:225] Handling call
0113 08:59:08.481023 (+ 1656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.487910 14267 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:08.483373 14441 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.545660 14351 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:08.593307 11960 rpcz_store.cc:276] 0113 08:59:06.871436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.441842 (+1570406us) service_pool.cc:225] Handling call
0113 08:59:08.539010 (+ 97168us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:08.594209 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 62075.9us
I20250113 08:59:08.594635 11966 rpcz_store.cc:276] 0113 08:59:07.494356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.455962 (+961606us) service_pool.cc:225] Handling call
0113 08:59:08.533700 (+ 77738us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.590409 14318 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.590057 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:08.478252 12721 rpcz_store.cc:269] 0113 08:59:06.514782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.596424 (+ 81642us) service_pool.cc:225] Handling call
0113 08:59:08.409548 (+1813124us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54006e1754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.409702 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":382572416}
I20250113 08:59:08.504022 14291 consensus_queue.cc:1046] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20250113 08:59:08.540649 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.663316 11968 raft_consensus.cc:2405] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:08.663899 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2136) took 1651 ms. Trace:
I20250113 08:59:08.664275 11968 rpcz_store.cc:276] 0113 08:59:07.012507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.518288 (+1505781us) service_pool.cc:225] Handling call
0113 08:59:08.663828 (+145540us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.515213 14406 consensus_queue.cc:1046] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20250113 08:59:08.561776 14393 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:08.593389 12684 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:08.560302 12719 rpcz_store.cc:269] 0113 08:59:06.442843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.460261 (+ 17418us) service_pool.cc:225] Handling call
0113 08:59:08.454004 (+1993743us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.608381 14444 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.608474 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2136) took 2006 ms. Trace:
I20250113 08:59:08.677708 11967 rpcz_store.cc:276] 0113 08:59:06.601984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.640303 (+ 38319us) service_pool.cc:225] Handling call
0113 08:59:06.711400 (+ 71097us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:08.503855 (+1792455us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:08.608212 (+104357us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b540015d454. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:08.608414 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3971755392}
W20250113 08:59:08.608812 14468 log.cc:926] Time spent T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20250113 08:59:08.608986 12686 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:08.609921 14297 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.610047 11931 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:08.610486 14129 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.610968 14449 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.611295 14461 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:08.586692 14333 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:08.614077 11884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d"
dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:43218
I20250113 08:59:08.765067 11884 raft_consensus.cc:604] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:08.765455 11884 raft_consensus.cc:612] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250113 08:59:08.592252 14446 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:08.502573 14320 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.621079 11892 raft_consensus.cc:2405] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:08.621419 11891 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:08.623937 11932 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:08.740741 14444 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.624958 12722 rpcz_store.cc:276] 0113 08:59:06.425183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.480416 (+2055233us) service_pool.cc:225] Handling call
0113 08:59:08.495897 (+ 15481us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:08.575931 (+ 80034us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:08.576224 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.597222 14360 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:08.598799 14342 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.617631 14315 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.621167 14141 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:08.593446 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 430) took 1791 ms. Trace:
I20250113 08:59:08.642498 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:08.646340 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 182831us
I20250113 08:59:08.650203 11885 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20250113 08:59:08.651639 11965 raft_consensus.cc:2405] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20250113 08:59:08.632033 14392 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:08.639843 12717 raft_consensus.cc:1235] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:08.659016 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:08.659807 11961 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:08.661559 11966 raft_consensus.cc:1235] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:08.662145 12007 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:08.640553 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 685) took 2184 ms. Trace:
I20250113 08:59:08.675998 12725 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:08.642468 14188 raft_consensus.cc:695] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:08.678777 11889 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:08.686285 11960 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:08.645465 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 551) took 2118 ms. Trace:
I20250113 08:59:08.739915 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.662497 14458 raft_consensus.cc:3049] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Stepping down as leader of term 2
W20250113 08:59:08.740777 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 130832us
I20250113 08:59:08.650826 14474 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.668365 14271 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:08.757642 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:08.661659 14473 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:59:08.772141 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 148345us
I20250113 08:59:08.591344 14273 raft_consensus.cc:695] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:08.747028 14291 raft_consensus.cc:3049] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:08.670022 14341 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:08.740804 14450 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.746759 14129 consensus_queue.cc:1046] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20250113 08:59:08.676606 14459 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.263s
I20250113 08:59:08.747298 14455 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:09.074191 11859 leader_election.cc:336] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.716s (SENT)
I20250113 08:59:09.074911 12720 rpcz_store.cc:276] 0113 08:59:06.454010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.476996 (+2022986us) service_pool.cc:225] Handling call
0113 08:59:08.477404 (+ 408us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:08.621994 (+144590us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:08.638418 (+ 16424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:09.075397 11859 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:09.075789 14458 raft_consensus.cc:738] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:08.765477 14461 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:08.838652 14464 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:09.078222 14461 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:09.078435 14464 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:08.817554 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 137831us
I20250113 08:59:09.079775 14297 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:09.080749 14464 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:09.081960 11965 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2133) took 2070 ms (client timeout 2491 ms). Trace:
I20250113 08:59:09.081844 14273 consensus_queue.cc:237] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:09.082640 11965 rpcz_store.cc:269] 0113 08:59:07.011493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.454459 (+1442966us) service_pool.cc:225] Handling call
0113 08:59:09.081902 (+627443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:09.083098 12724 raft_consensus.cc:2405] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:09.084312 11965 raft_consensus.cc:1235] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:09.084932 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2119) took 2246 ms. Trace:
I20250113 08:59:08.810415 14351 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:08.995677 14358 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:09.082661 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2137) took 2480 ms. Trace:
I20250113 08:59:09.124801 11966 rpcz_store.cc:276] 0113 08:59:06.602322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.658138 (+2055816us) service_pool.cc:225] Handling call
0113 08:59:08.658490 (+ 352us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.082400 (+423910us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.082600 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:09.127465 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45204.6us
I20250113 08:59:09.085871 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 428) took 2309 ms. Trace:
I20250113 08:59:09.076977 11963 rpcz_store.cc:276] 0113 08:59:06.801897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.455438 (+1653541us) service_pool.cc:225] Handling call
0113 08:59:08.593379 (+137941us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:09.077669 14291 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:09.084820 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:08.968204 14466 log.cc:926] Time spent T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.311s user 0.001s sys 0.000s
I20250113 08:59:08.759313 14467 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:09.076517 14488 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:09.086753 14315 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:09.086951 12718 rpcz_store.cc:276] 0113 08:59:06.523614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.502961 (+1979347us) service_pool.cc:225] Handling call
0113 08:59:08.503355 (+ 394us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:08.620781 (+117426us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:08.642014 (+ 21233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:08.846911 14481 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
W20250113 08:59:08.951995 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 358659us
I20250113 08:59:09.089347 14123 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:09.089630 11962 raft_consensus.cc:1235] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:09.033455 14352 consensus_meta.cc:287] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.589s user 0.003s sys 0.004s
I20250113 08:59:08.973579 14441 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:09.091466 14311 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:08.823369 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250113 08:59:09.160821 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:59:09.097659 14473 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:08.985512 14421 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.975345 14318 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:08.924027 14212 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:08.978006 14314 consensus_meta.cc:287] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.640s user 0.012s sys 0.000s
I20250113 08:59:09.098781 14242 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:08.809571 14449 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:09.138985 14271 consensus_queue.cc:1046] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20250113 08:59:09.099771 11934 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:09.100572 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.077359 14458 consensus_queue.cc:260] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:09.078091 11885 pending_rounds.cc:77] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 6
I20250113 08:59:09.176932 11885 pending_rounds.cc:99] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20250113 08:59:08.988593 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.177273 14458 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:09.157014 14391 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:08.771025 14446 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:08.969364 14478 log.cc:926] Time spent T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
W20250113 08:59:08.843742 14463 log.cc:926] Time spent T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
W20250113 08:59:08.881955 14465 log.cc:926] Time spent T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.307s user 0.002s sys 0.000s
I20250113 08:59:09.037081 14211 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:08.989701 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2132) took 2398 ms. Trace:
I20250113 08:59:08.979640 14265 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:59:08.952716 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 359433us
I20250113 08:59:09.077931 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 781) took 2472 ms. Trace:
I20250113 08:59:08.774518 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.060716 14393 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:09.085386 14474 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.108372 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:09.110118 14460 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.008s sys 0.000s
W20250113 08:59:09.111107 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 510806us
I20250113 08:59:09.086337 14188 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.076813 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.115043 14169 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:09.075397 11931 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:09.098621 11968 raft_consensus.cc:2388] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:09.118496 11965 rpcz_store.cc:276] 0113 08:59:06.838996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.083842 (+2244846us) service_pool.cc:225] Handling call
0113 08:59:09.084149 (+ 307us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.084705 (+ 556us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.084863 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:09.129561 11960 rpcz_store.cc:276] 0113 08:59:06.776883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.658730 (+1881847us) service_pool.cc:225] Handling call
0113 08:59:08.685824 (+ 27094us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.085558 (+399734us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.085800 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:09.133857 11963 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:09.134557 12724 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:59:08.824803 14325 consensus_meta.cc:287] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.545s user 0.007s sys 0.004s
I20250113 08:59:08.768821 14333 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:09.142234 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 554) took 2593 ms. Trace:
I20250113 08:59:09.092285 14341 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:09.128093 14351 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:09.150759 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 488671us
W20250113 08:59:09.155155 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 49740us
I20250113 08:59:09.157511 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:09.158248 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:09.162770 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2122) took 2196 ms. Trace:
I20250113 08:59:09.140794 14129 consensus_queue.cc:1225] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:09.162460 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:09.167415 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:09.169934 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:09.171720 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:08.991421 14407 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.172344 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:09.155205 14123 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.162173 14473 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:09.174050 14449 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.162204 14409 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:09.171792 14314 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:09.164956 14311 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.171991 14212 consensus_queue.cc:1035] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250113 08:59:09.134336 14444 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.146792 14464 consensus_queue.cc:1046] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250113 08:59:09.146385 14141 consensus_queue.cc:1046] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.023s
I20250113 08:59:09.306164 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:09.223809 12723 raft_consensus.cc:2388] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:09.305738 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:09.588045 12040 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:09.178776 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:59:09.185575 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:09.177291 11885 raft_consensus.cc:2883] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:09.581760 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:09.594079 11962 rpcz_store.cc:276] 0113 08:59:06.959024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.089157 (+2130133us) service_pool.cc:225] Handling call
0113 08:59:09.089472 (+ 315us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.154654 (+ 65182us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.154907 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:09.586930 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2127) took 2577 ms. Trace:
I20250113 08:59:09.197938 12006 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:09.596347 12007 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:09.461006 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:09.597589 12038 raft_consensus.cc:2388] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:09.595845 12034 raft_consensus.cc:2371] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4. Re-sending same reply.
I20250113 08:59:09.161660 14342 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:09.587797 12008 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:09.586272 12726 rpcz_store.cc:276] 0113 08:59:06.548603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.529671 (+1981068us) service_pool.cc:225] Handling call
0113 08:59:08.558430 (+ 28759us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:08.558709 (+ 279us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:08.677699 (+118990us) log.cc:844] Serialized 14 byte log entry
0113 08:59:09.140839 (+463140us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:09.140994 (+ 155us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:09.141211 (+ 217us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.141426 (+ 215us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:09.141970 (+ 544us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14100,"prepare.queue_time_us":92122,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":9785,"raft.queue_time_us":758,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":5773,"thread_start_us":545776,"threads_started":2,"wal-append.queue_time_us":737689}
I20250113 08:59:09.365661 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:09.231585 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 131883us
I20250113 08:59:09.625466 11889 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 9: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:09.163455 14481 raft_consensus.cc:695] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:09.166942 14435 consensus_queue.cc:1046] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.554s
I20250113 08:59:09.167527 14358 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.583s
I20250113 08:59:09.156818 14467 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:09.625706 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 550352us
I20250113 08:59:09.953425 11960 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:09.954802 11931 leader_election.cc:400] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
W20250113 08:59:09.526896 14454 log.cc:926] Time spent T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250113 08:59:09.954905 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2128) took 2945 ms. Trace:
I20250113 08:59:09.169556 14488 consensus_queue.cc:1046] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.787s
I20250113 08:59:09.722187 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1754) took 1204 ms. Trace:
I20250113 08:59:09.957341 11885 rpcz_store.cc:276] 0113 08:59:08.518021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.629743 (+111722us) service_pool.cc:225] Handling call
0113 08:59:08.649808 (+ 20065us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.721822 (+1072014us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.722101 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:09.957892 12034 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250113 08:59:09.958421 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:09.959192 11885 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:09.958568 14291 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:09.960575 14311 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:09.963199 14314 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:09.702677 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 106407us
I20250113 08:59:09.965747 14449 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:09.645632 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 57955.2us
I20250113 08:59:09.981312 12008 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:09.702358 14450 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:09.361723 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:09.701644 11887 raft_consensus.cc:2388] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:09.734107 12039 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 3: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:09.732419 12042 raft_consensus.cc:2405] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:09.971300 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.896147 12040 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:09.967957 14407 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:09.956950 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:09.965406 14481 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:10.014905 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 60250.3us
I20250113 08:59:10.015661 12034 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250113 08:59:10.016263 11931 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:09.750963 12035 raft_consensus.cc:1235] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:09.787192 11968 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:09.624496 12033 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:09.455981 14421 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:09.531131 14242 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:09.966164 12007 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:09.627481 11888 raft_consensus.cc:2388] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:09.752684 11884 raft_consensus.cc:3049] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:10.026278 11884 raft_consensus.cc:738] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:09.790517 14441 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:09.454879 14315 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:09.602805 14351 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:09.997210 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 701) took 1226 ms. Trace:
I20250113 08:59:09.731863 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.039204 12722 rpcz_store.cc:276] 0113 08:59:08.774129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.627028 (+852899us) service_pool.cc:225] Handling call
0113 08:59:09.997149 (+370121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.039534 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 692) took 1407 ms. Trace:
W20250113 08:59:09.754734 12723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 558) took 2508 ms (client timeout 2127 ms). Trace:
I20250113 08:59:09.602082 14391 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:09.754396 14137 raft_consensus.cc:2743] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:09.732554 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 534716us
I20250113 08:59:10.000279 11885 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:59:09.827924 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 46343.8us
I20250113 08:59:09.645169 12719 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:09.811115 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.732100 14273 raft_consensus.cc:3049] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:09.802542 14211 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:09.732208 14438 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:09.833070 14477 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:09.849740 12720 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 9: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:09.285851 14267 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:09.701637 14342 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:09.178777 14318 consensus_queue.cc:1046] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.830s
I20250113 08:59:09.178306 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.646553 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:09.645488 14409 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:09.285545 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:09.596115 11959 rpcz_store.cc:276] 0113 08:59:06.591183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:06.593368 (+ 2185us) service_pool.cc:225] Handling call
0113 08:59:08.301353 (+1707985us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:08.418201 (+116848us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:08.989395 (+571194us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b5400196250. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:08.989594 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3927108992}
I20250113 08:59:09.270885 14150 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:10.014300 11839 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20250113 08:59:09.603210 14460 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.998s user 0.018s sys 0.004s
I20250113 08:59:10.019209 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2129) took 3009 ms. Trace:
I20250113 08:59:09.752823 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2130) took 2742 ms. Trace:
I20250113 08:59:09.596391 11963 rpcz_store.cc:276] 0113 08:59:07.009333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.128750 (+2119417us) service_pool.cc:225] Handling call
0113 08:59:09.129135 (+ 385us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.586677 (+457542us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.586873 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:09.847052 14344 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s user 0.012s sys 0.005s
I20250113 08:59:09.956094 11960 rpcz_store.cc:276] 0113 08:59:07.009701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.622868 (+2613167us) service_pool.cc:225] Handling call
0113 08:59:09.952744 (+329876us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.954407 (+ 1663us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.954812 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.029951 11891 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:10.031400 12034 raft_consensus.cc:1235] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:10.021574 14212 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250113 08:59:10.040820 12723 rpcz_store.cc:269] 0113 08:59:07.245968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.153329 (+907361us) service_pool.cc:225] Handling call
0113 08:59:09.754473 (+1601144us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b54006bdad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:09.754639 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2074689280}
I20250113 08:59:10.039908 12726 rpcz_store.cc:276] 0113 08:59:08.632496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.702064 (+1069568us) service_pool.cc:225] Handling call
0113 08:59:10.039474 (+337410us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.043294 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.178629 12040 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:10.017596 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 699) took 1270 ms. Trace:
I20250113 08:59:10.049017 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:09.534781 14392 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:09.592867 12717 rpcz_store.cc:276] 0113 08:59:06.605958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.533767 (+1927809us) service_pool.cc:225] Handling call
0113 08:59:08.622167 (+ 88400us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.077682 (+455515us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.077876 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.027146 11884 consensus_queue.cc:260] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.054780 12685 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:10.210184 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 155488us
I20250113 08:59:10.208302 12722 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:10.036326 14441 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.064160 12724 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:10.215529 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.216102 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 696) took 1571 ms. Trace:
I20250113 08:59:10.216450 12724 rpcz_store.cc:276] 0113 08:59:08.644724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.623042 (+978318us) service_pool.cc:225] Handling call
0113 08:59:10.216043 (+593001us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.067358 14211 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:10.076467 11833 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.BulkChangeConfig -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 172803us
I20250113 08:59:10.040319 14265 raft_consensus.cc:2743] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:09.632457 14325 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.086515 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.088034 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.093839 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:10.102465 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 69049.1us
W20250113 08:59:10.102931 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 191307us
I20250113 08:59:10.104583 14291 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:10.105280 14315 consensus_queue.cc:1035] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250113 08:59:10.110464 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 515466us
I20250113 08:59:10.063252 14497 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:10.050873 14391 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:10.146327 11931 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:10.265599 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 119355us
I20250113 08:59:10.266427 11931 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.268116 11931 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:10.147607 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 181553us
I20250113 08:59:10.055287 14273 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:10.269477 14518 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:10.270417 14518 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:10.270094 14273 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.270820 14518 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.271287 14518 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:10.063899 14438 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.271979 14438 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:10.072050 14477 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.183281 12718 rpcz_store.cc:276] 0113 08:59:08.746864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.166156 (+419292us) service_pool.cc:225] Handling call
0113 08:59:10.017534 (+851378us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.273571 14477 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:10.274212 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.274875 12718 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:10.275796 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.276387 12718 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:10.161331 12036 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:10.171833 14150 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.067781 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 695) took 1426 ms. Trace:
I20250113 08:59:10.067317 14242 raft_consensus.cc:3049] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:10.167721 11968 rpcz_store.cc:276] 0113 08:59:07.010109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.623854 (+2613745us) service_pool.cc:225] Handling call
0113 08:59:09.750587 (+126733us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.018786 (+268199us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.019128 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.168393 11965 rpcz_store.cc:276] 0113 08:59:07.010437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.622961 (+2612524us) service_pool.cc:225] Handling call
0113 08:59:09.752158 (+129197us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.752378 (+ 220us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:09.752415 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:09.752470 (+ 55us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:09.752540 (+ 70us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:09.752604 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:09.752751 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.170490 11963 raft_consensus.cc:1270] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:59:10.173530 12684 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.228600 14474 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:10.181411 12041 raft_consensus.cc:2388] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:10.146131 14358 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:10.182632 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:10.145642 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 700) took 1388 ms (client timeout 1631 ms). Trace:
I20250113 08:59:10.220141 14441 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:10.145960 14267 raft_consensus.cc:695] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:10.185810 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.341938 12034 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:59:10.208604 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 99703.3us
I20250113 08:59:10.177875 14393 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:10.166818 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 694) took 1531 ms. Trace:
I20250113 08:59:10.016904 14450 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.217509 11960 raft_consensus.cc:1270] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:59:10.349361 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2139) took 3300 ms. Trace:
I20250113 08:59:10.349872 11960 rpcz_store.cc:276] 0113 08:59:07.049235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.172562 (+3123327us) service_pool.cc:225] Handling call
0113 08:59:10.210072 (+ 37510us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.348956 (+138884us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.349283 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.350579 11961 raft_consensus.cc:1235] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:10.217897 12008 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:10.351899 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 134104us
I20250113 08:59:10.352201 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:10.353559 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.185514 14123 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:10.221532 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.017s sys 0.058s
I20250113 08:59:10.177035 14212 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.361697 14212 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:10.362471 14358 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:10.362877 14358 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.363206 14358 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:10.363837 14488 consensus_queue.cc:1046] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:10.364945 14488 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:10.252223 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:10.252976 12686 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:10.240649 12039 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:10.262272 14497 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:10.263521 14513 log.cc:926] Time spent T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250113 08:59:10.371315 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.264117 14311 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:10.271325 14273 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:10.220081 14352 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:10.112741 14449 raft_consensus.cc:3049] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:10.287398 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:10.379683 12726 raft_consensus.cc:2371] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5. Re-sending same reply.
I20250113 08:59:10.288327 12033 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9.
I20250113 08:59:10.293231 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.295965 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2132) took 3284 ms. Trace:
I20250113 08:59:10.272759 14351 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:10.282596 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2138) took 3246 ms. Trace:
W20250113 08:59:10.283215 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 109759us
I20250113 08:59:10.280778 11890 raft_consensus.cc:2388] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:10.278198 11931 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.296469 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:10.340921 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:10.341836 11965 raft_consensus.cc:1235] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:10.283965 14455 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:10.229053 14467 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:10.355484 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.277817 14521 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250113 08:59:10.358494 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.867s user 0.017s sys 0.058s
I20250113 08:59:10.361155 11960 raft_consensus.cc:1270] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250113 08:59:10.281772 14150 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:10.215704 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:10.529647 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2140) took 3064 ms. Trace:
I20250113 08:59:10.278048 14444 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:10.292574 14211 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:10.279803 14392 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:10.281155 14242 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:10.277221 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:10.382290 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:10.382961 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.269479 12007 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:10.354893 14342 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:10.369989 14465 log.cc:926] Time spent T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20250113 08:59:10.561336 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 291917us
I20250113 08:59:10.562356 12007 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.510715 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2179) took 1322 ms. Trace:
I20250113 08:59:10.563719 12033 rpcz_store.cc:276] 0113 08:59:09.188547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.270686 (+ 82139us) service_pool.cc:225] Handling call
0113 08:59:10.501966 (+1231280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.512101 11891 raft_consensus.cc:2405] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:10.382333 12724 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:10.280953 12719 rpcz_store.cc:276] 0113 08:59:08.641045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.746626 (+105581us) service_pool.cc:225] Handling call
0113 08:59:10.065810 (+1319184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.568178 12035 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250113 08:59:10.568477 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.514266 12034 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250113 08:59:10.569592 12035 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:59:10.570243 12034 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250113 08:59:10.287598 12720 rpcz_store.cc:269] 0113 08:59:08.757169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.076093 (+318924us) service_pool.cc:225] Handling call
0113 08:59:10.140597 (+1064504us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.573758 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.280277 14438 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:10.381421 11962 rpcz_store.cc:276] 0113 08:59:07.011201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.624347 (+2613146us) service_pool.cc:225] Handling call
0113 08:59:09.750102 (+125755us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:09.847427 (+ 97325us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:09.847470 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:09.969937 (+122467us) log.cc:844] Serialized 52 byte log entry
0113 08:59:10.165770 (+195833us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:10.165818 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.165945 (+ 127us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:10.295694 (+129749us) raft_consensus.cc:1680] finished
0113 08:59:10.295721 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:10.295895 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":158610,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":135,"thread_start_us":195374,"threads_started":1,"wal-append.queue_time_us":251054}
I20250113 08:59:10.517221 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.518517 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2174) took 1336 ms. Trace:
W20250113 08:59:10.526921 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 248802us
I20250113 08:59:10.530145 11960 rpcz_store.cc:276] 0113 08:59:07.465761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.351082 (+2885321us) service_pool.cc:225] Handling call
0113 08:59:10.357953 (+ 6871us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.526085 (+168132us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.529564 (+ 3479us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:10.532511 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 279662us
I20250113 08:59:10.579552 12041 rpcz_store.cc:276] 0113 08:59:09.182751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.182861 (+ 110us) service_pool.cc:225] Handling call
0113 08:59:10.518266 (+1335405us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540002cbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:10.518441 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":382956800}
I20250113 08:59:10.580726 12686 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:10.580039 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.582346 12686 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:10.533216 14458 consensus_meta.cc:287] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.355s user 0.013s sys 0.000s
I20250113 08:59:10.535792 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:10.586506 12007 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.531646 14318 raft_consensus.cc:3049] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
W20250113 08:59:10.539453 14446 consensus_meta.cc:287] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.864s user 0.007s sys 0.003s
I20250113 08:59:10.544622 11967 raft_consensus.cc:2388] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:10.348778 12721 rpcz_store.cc:276] 0113 08:59:08.633126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.178001 (+544875us) service_pool.cc:225] Handling call
0113 08:59:10.158382 (+980381us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.376840 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:10.513163 14333 consensus_meta.cc:287] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.865s user 0.009s sys 0.002s
I20250113 08:59:10.379391 14449 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:10.383525 14406 raft_consensus.cc:2758] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250113 08:59:10.508183 14421 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:10.592828 14188 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 08:59:10.553891 11889 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250113 08:59:10.514667 14409 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:10.558370 11885 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:10.559396 11968 raft_consensus.cc:1235] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:10.617228 14271 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.004s sys 0.023s
I20250113 08:59:10.567579 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.571455 12034 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20250113 08:59:10.572189 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 218466us
I20250113 08:59:10.547571 14344 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:10.501011 11963 rpcz_store.cc:276] 0113 08:59:07.035834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.169758 (+3133924us) service_pool.cc:225] Handling call
0113 08:59:10.170110 (+ 352us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.282319 (+112209us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.282537 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.574608 12723 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:10.375454 14488 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.595950 11887 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:10.597046 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:10.592972 11893 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.454s user 0.012s sys 0.032s
I20250113 08:59:10.600271 11960 raft_consensus.cc:1270] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:59:10.599722 12007 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:10.601955 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1905) took 1445 ms. Trace:
I20250113 08:59:10.577445 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2137) took 3564 ms. Trace:
I20250113 08:59:10.618966 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2140) took 3228 ms. Trace:
I20250113 08:59:10.619216 11966 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:10.618966 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2145) took 1958 ms (client timeout 1960 ms). Trace:
W20250113 08:59:10.679646 11967 rpcz_store.cc:269] 0113 08:59:08.660062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.690438 (+ 30376us) service_pool.cc:225] Handling call
0113 08:59:10.618737 (+1928299us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400196754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:10.618903 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3321171840}
I20250113 08:59:10.586966 14392 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:10.589668 14522 raft_consensus.cc:3049] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:10.629554 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2124) took 3621 ms. Trace:
I20250113 08:59:10.636668 14315 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:10.595088 14467 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:10.580843 14211 consensus_queue.cc:1046] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20250113 08:59:10.607308 14444 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:10.566257 12033 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:10.640976 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:10.579399 14481 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.008s sys 0.019s
I20250113 08:59:10.642002 12008 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.597816 14318 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:10.589680 14525 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:10.589751 14476 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:10.650869 11890 rpcz_store.cc:276] 0113 08:59:09.156669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.156774 (+ 105us) service_pool.cc:225] Handling call
0113 08:59:10.601676 (+1444902us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b540038f254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:10.601878 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1574544768}
I20250113 08:59:10.649228 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:10.650067 11963 raft_consensus.cc:1235] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:10.649852 11893 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.511s user 0.013s sys 0.032s
I20250113 08:59:10.650280 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2142) took 2499 ms. Trace:
I20250113 08:59:10.726439 12034 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:10.726619 11960 rpcz_store.cc:276] 0113 08:59:08.151054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.579874 (+2428820us) service_pool.cc:225] Handling call
0113 08:59:10.599807 (+ 19933us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.650042 (+ 50235us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.650229 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:10.671663 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 72372.1us
I20250113 08:59:10.679401 11965 rpcz_store.cc:276] 0113 08:59:07.390611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.282383 (+2891772us) service_pool.cc:225] Handling call
0113 08:59:10.282728 (+ 345us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.509282 (+226554us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.618737 (+109455us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b54001ae0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:10.618904 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135272704}
I20250113 08:59:10.682807 11891 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:59:10.644673 14271 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.931s user 0.005s sys 0.023s
I20250113 08:59:10.687798 11961 rpcz_store.cc:276] 0113 08:59:07.008163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.103076 (+2094913us) service_pool.cc:225] Handling call
0113 08:59:09.103404 (+ 328us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.359894 (+1256490us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.629214 (+269320us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400050ad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:10.629473 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2856414080}
I20250113 08:59:10.570163 12718 raft_consensus.cc:2371] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9. Re-sending same reply.
I20250113 08:59:10.627997 14526 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:10.562379 14267 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.642081 14458 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:10.696841 11889 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 9. (term mismatch)
I20250113 08:59:10.816293 11889 pending_rounds.cc:77] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 8
I20250113 08:59:10.727918 14467 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:10.725227 14497 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:10.644248 14344 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.649030 14311 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:10.651121 14406 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:10.615773 14446 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:10.719491 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 77617.4us
I20250113 08:59:10.838111 12008 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:10.824705 14520 log.cc:926] Time spent T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.296s user 0.000s sys 0.001s
I20250113 08:59:10.672659 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.687932 14342 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:10.634078 14333 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.731916 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2143) took 2383 ms. Trace:
W20250113 08:59:10.732715 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 222305us
I20250113 08:59:10.733335 12724 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250113 08:59:10.692699 14522 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:10.736886 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:10.737498 12726 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20250113 08:59:10.718317 14481 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.009s sys 0.020s
I20250113 08:59:10.698220 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.862269 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:10.739396 12684 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.643982 14421 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.716034 14450 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:10.798781 14474 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20250113 08:59:10.718384 14455 raft_consensus.cc:695] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:10.635025 14242 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.816654 11889 pending_rounds.cc:99] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 6.9
I20250113 08:59:10.874938 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:10.875154 11889 raft_consensus.cc:2883] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:10.875648 11888 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:10.698561 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.846758 14342 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.644897 14527 consensus_queue.cc:1046] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.099s
I20250113 08:59:10.819730 11960 raft_consensus.cc:1235] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:10.860025 12717 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:10.740635 14325 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:10.887682 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.888101 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 499) took 1286 ms. Trace:
I20250113 08:59:10.888477 12723 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 9: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:59:10.825134 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 293680us
I20250113 08:59:10.741314 14352 raft_consensus.cc:695] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:10.800390 14123 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:10.608408 14521 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20250113 08:59:10.707110 14315 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.816435 14211 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:10.847033 11963 rpcz_store.cc:276] 0113 08:59:08.348341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.644801 (+2296460us) service_pool.cc:225] Handling call
0113 08:59:10.645146 (+ 345us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.731651 (+ 86505us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.731860 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.738274 14444 consensus_queue.cc:1046] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20250113 08:59:10.850566 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 782) took 4150 ms. Trace:
I20250113 08:59:10.827801 14409 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:10.851469 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 784) took 4049 ms. Trace:
W20250113 08:59:10.911473 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 172150us
I20250113 08:59:10.912016 12726 rpcz_store.cc:276] 0113 08:59:06.801603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.718213 (+3916610us) service_pool.cc:225] Handling call
0113 08:59:10.737106 (+ 18893us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.849431 (+112325us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.851413 (+ 1982us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.851868 12685 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:10.833598 14344 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:10.747942 14392 consensus_queue.cc:1046] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20250113 08:59:10.859506 11964 raft_consensus.cc:2388] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:10.856141 12722 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:59:10.801443 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 348363us
I20250113 08:59:10.866777 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2126) took 3857 ms. Trace:
I20250113 08:59:10.867775 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2138) took 3756 ms. Trace:
I20250113 08:59:10.797515 14526 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.746044 14476 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.716272 14449 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.815562 14341 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20250113 08:59:10.842944 14271 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:10.878142 12008 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:10.846647 14318 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.893010 12721 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:10.822710 14458 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:10.732344 14525 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:10.679109 11959 rpcz_store.cc:276] 0113 08:59:07.012815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.177107 (+3164292us) service_pool.cc:225] Handling call
0113 08:59:10.257833 (+ 80726us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.514777 (+256944us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:10.526682 (+ 11905us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:10.541008 (+ 14326us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:10.548126 (+ 7118us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.552631 (+ 4505us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:10.572603 (+ 19972us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":138173,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":33170}
I20250113 08:59:10.928670 14481 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:10.922273 14539 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:10.888633 11886 rpcz_store.cc:276] 0113 08:59:09.601598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.624206 (+ 22608us) service_pool.cc:225] Handling call
0113 08:59:10.602486 (+978280us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.873501 (+271015us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:10.873532 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:10.873574 (+ 42us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:10.873617 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.873680 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:10.887941 (+ 14261us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x7b540038f250. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:10.888047 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5073,"raft.queue_time_us":258880,"raft.run_wall_time_us":20279,"spinlock_wait_cycles":1870590848,"thread_start_us":265408,"threads_started":1}
I20250113 08:59:10.865893 14438 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:10.865602 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.841320 14406 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.842317 14311 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.905802 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:10.907079 12724 rpcz_store.cc:276] 0113 08:59:06.699844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.705864 (+4006020us) service_pool.cc:225] Handling call
0113 08:59:10.733005 (+ 27141us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:10.850232 (+117227us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.850491 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.883114 14537 raft_consensus.cc:3049] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:10.883036 14342 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:10.914939 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 63141.8us
I20250113 08:59:10.916931 12684 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.746009 14351 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:10.922434 11968 rpcz_store.cc:276] 0113 08:59:07.111642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.281491 (+3169849us) service_pool.cc:225] Handling call
0113 08:59:10.281888 (+ 397us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.640975 (+359087us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.867521 (+226546us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54002f0b54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:10.867708 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":637667200}
I20250113 08:59:10.863147 12034 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
W20250113 08:59:10.923413 11859 leader_election.cc:336] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.691s (SENT)
I20250113 08:59:10.961364 11859 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:10.961861 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 44963.5us
W20250113 08:59:10.962661 11859 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.573s (SENT) [suppressed 14 similar messages]
I20250113 08:59:10.921941 11966 rpcz_store.cc:276] 0113 08:59:07.008929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.125928 (+2116999us) service_pool.cc:225] Handling call
0113 08:59:10.619042 (+1493114us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.682189 (+ 63147us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.866573 (+184384us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400196750. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:10.866716 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3391197440}
W20250113 08:59:10.907434 14461 consensus_meta.cc:287] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.829s user 0.014s sys 0.002s
W20250113 08:59:10.909312 14523 log.cc:926] Time spent T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.301s user 0.001s sys 0.000s
I20250113 08:59:10.924701 11932 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:10.933669 12722 raft_consensus.cc:1235] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:10.906541 14211 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:10.934062 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 56013.8us
I20250113 08:59:10.882766 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2147) took 1801 ms. Trace:
I20250113 08:59:10.970667 11960 rpcz_store.cc:276] 0113 08:59:09.081547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.727715 (+1646168us) service_pool.cc:225] Handling call
0113 08:59:10.816992 (+ 89277us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.882502 (+ 65510us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.882694 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.971189 12008 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:10.939489 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1758) took 2394 ms. Trace:
I20250113 08:59:10.899922 14497 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20250113 08:59:10.935397 14271 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:10.923188 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 687) took 4133 ms. Trace:
W20250113 08:59:10.932651 14473 consensus_meta.cc:287] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.107s user 0.011s sys 0.000s
I20250113 08:59:10.999398 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 688) took 2746 ms. Trace:
I20250113 08:59:10.999866 12722 rpcz_store.cc:276] 0113 08:59:08.226041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.931819 (+2705778us) service_pool.cc:225] Handling call
0113 08:59:10.932186 (+ 367us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.972262 (+ 40076us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.972447 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:10.965620 14532 log.cc:926] Time spent T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.243s user 0.001s sys 0.000s
I20250113 08:59:10.936290 14242 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:10.959059 12685 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:10.958353 12723 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:10.963650 11859 leader_election.cc:336] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 2.028s (SENT)
I20250113 08:59:10.940160 14474 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:10.914108 14540 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20250113 08:59:10.949779 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:10.950966 14318 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:10.967032 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:11.034679 12034 raft_consensus.cc:2388] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:10.952898 14515 consensus_queue.cc:1035] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20250113 08:59:10.704960 11967 raft_consensus.cc:1270] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250113 08:59:10.968801 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 223907us
W20250113 08:59:10.966482 14391 consensus_meta.cc:287] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.701s user 0.006s sys 0.006s
I20250113 08:59:10.971767 11893 rpcz_store.cc:276] 0113 08:59:08.545519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.661630 (+116111us) service_pool.cc:225] Handling call
0113 08:59:08.752189 (+ 90559us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:08.752475 (+ 286us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:09.138504 (+386029us) log.cc:844] Serialized 14 byte log entry
0113 08:59:10.939152 (+1800648us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:10.939212 (+ 60us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:10.939262 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.939323 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:10.939432 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2341,"prepare.queue_time_us":215333,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":167713,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":9486,"thread_start_us":2125964,"threads_started":3,"wal-append.queue_time_us":1579978}
W20250113 08:59:10.998540 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 73928.2us
I20250113 08:59:10.902856 14123 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:10.954355 14539 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:10.955646 14449 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:11.007120 11859 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:10.957731 14537 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:11.010372 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 114843us
I20250113 08:59:11.012725 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 690) took 2748 ms. Trace:
I20250113 08:59:11.042282 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:11.027742 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 68783.6us
I20250113 08:59:10.997850 14211 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:10.967621 12724 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20250113 08:59:11.037140 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 691) took 2750 ms. Trace:
W20250113 08:59:11.075233 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 112673us
I20250113 08:59:11.008989 14392 raft_consensus.cc:3049] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:11.011755 14351 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:10.952450 14455 consensus_queue.cc:237] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:11.041769 14531 log.cc:926] Time spent T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
I20250113 08:59:11.040727 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2146) took 1959 ms. Trace:
W20250113 08:59:11.027823 14534 log.cc:926] Time spent T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20250113 08:59:11.008100 14488 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:11.073408 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 487806us
I20250113 08:59:11.074154 12723 rpcz_store.cc:276] 0113 08:59:08.263999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.957842 (+2693843us) service_pool.cc:225] Handling call
0113 08:59:10.958182 (+ 340us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.012378 (+ 54196us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.012650 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:10.959775 14352 consensus_queue.cc:237] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.098003 12686 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:11.075320 12724 pending_rounds.cc:77] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting all ops after (but not including) 4
I20250113 08:59:11.098676 12724 pending_rounds.cc:99] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250113 08:59:11.084357 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:11.099112 12724 raft_consensus.cc:2883] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:11.030918 14446 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:10.972990 14461 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:11.040982 14212 consensus_meta.cc:287] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.679s user 0.005s sys 0.006s
I20250113 08:59:10.970883 14527 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20250113 08:59:11.088973 11967 rpcz_store.cc:276] 0113 08:59:09.081181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.680680 (+1599499us) service_pool.cc:225] Handling call
0113 08:59:10.704540 (+ 23860us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.040468 (+335928us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.040668 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:11.108870 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:11.071852 14421 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:11.113667 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:11.075436 12722 rpcz_store.cc:276] 0113 08:59:08.272901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.000844 (+2727943us) service_pool.cc:225] Handling call
0113 08:59:11.022565 (+ 21721us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.022809 (+ 244us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:11.022843 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:11.022898 (+ 55us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:11.022941 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.023010 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:11.023148 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:11.035041 14544 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:11.044620 12720 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:11.082263 14392 raft_consensus.cc:738] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:10.999255 14545 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.122s
W20250113 08:59:11.089529 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 118412us
I20250113 08:59:11.026307 14460 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:11.003516 12717 rpcz_store.cc:276] 0113 08:59:06.789601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.720917 (+3931316us) service_pool.cc:225] Handling call
0113 08:59:10.837387 (+116470us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:10.891891 (+ 54504us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:10.918933 (+ 27042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:11.112593 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.117352 12686 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:11.118160 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 689) took 2888 ms. Trace:
I20250113 08:59:11.162446 12724 rpcz_store.cc:276] 0113 08:59:08.229697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.955971 (+2726274us) service_pool.cc:225] Handling call
0113 08:59:10.967256 (+ 11285us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.117920 (+150664us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.118096 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:11.070667 14539 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:11.116847 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 153245us
I20250113 08:59:11.088325 14351 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:11.006958 14473 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:11.124410 14315 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.124631 14311 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:11.132390 14273 consensus_meta.cc:287] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.752s user 0.008s sys 0.001s
I20250113 08:59:11.135118 14406 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:11.135623 14518 consensus_meta.cc:287] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.864s user 0.010s sys 0.000s
I20250113 08:59:11.112712 14212 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.183562 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.137428 14421 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:11.098723 14474 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250113 08:59:11.138103 14407 consensus_meta.cc:287] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.120s user 0.006s sys 0.006s
I20250113 08:59:11.118126 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:11.140344 14358 consensus_meta.cc:287] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.777s user 0.007s sys 0.002s
I20250113 08:59:11.117717 14543 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.141676 14458 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:11.195122 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.142778 14549 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:11.082110 14548 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:11.148284 12725 raft_consensus.cc:2388] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:59:11.112862 14477 consensus_meta.cc:287] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.839s user 0.010s sys 0.003s
I20250113 08:59:11.070919 14391 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:11.179611 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 62363.2us
I20250113 08:59:11.186951 14421 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:11.180192 14515 consensus_queue.cc:1035] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:59:11.041149 14536 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20250113 08:59:11.209293 14541 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.209849 14541 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:11.202220 14526 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.213557 14559 consensus_queue.cc:1046] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250113 08:59:11.217937 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2141) took 3067 ms (client timeout 2725 ms). Trace:
W20250113 08:59:11.218370 11964 rpcz_store.cc:269] 0113 08:59:08.150759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.526046 (+375287us) service_pool.cc:225] Handling call
0113 08:59:11.217713 (+2691667us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b54001a1dd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:11.217860 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5153794944}
I20250113 08:59:11.182346 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 563) took 2727 ms. Trace:
W20250113 08:59:11.184687 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 103145us
W20250113 08:59:11.161484 14441 consensus_meta.cc:287] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.812s user 0.012s sys 0.000s
I20250113 08:59:11.220542 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.220402 14441 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:11.182833 12034 raft_consensus.cc:2405] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:11.188004 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:11.188196 12038 raft_consensus.cc:1235] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:11.112267 14533 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.172260 14351 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:11.173044 14552 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:11.200784 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.203444 12039 raft_consensus.cc:2363] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6.
W20250113 08:59:11.172513 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 104701us
I20250113 08:59:11.205471 11856 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:11.205855 11962 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:11.182140 14537 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.207765 14527 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:59:11.211830 14541 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:11.221128 11859 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:11.221556 12037 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:11.221726 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.255088 11885 raft_consensus.cc:2388] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:11.255010 11859 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:11.186365 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 110394us
I20250113 08:59:11.225077 12006 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:11.220134 12723 rpcz_store.cc:276] 0113 08:59:08.454907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.097657 (+2642750us) service_pool.cc:225] Handling call
0113 08:59:11.098026 (+ 369us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.098274 (+ 248us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:11.116069 (+ 17795us) log.cc:844] Serialized 14 byte log entry
0113 08:59:11.181729 (+ 65660us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:11.181817 (+ 88us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:11.181921 (+ 104us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.182046 (+ 125us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:11.182265 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1956,"prepare.queue_time_us":18042,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":16775,"raft.queue_time_us":11285,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":5577,"thread_start_us":62678,"threads_started":1,"wal-append.queue_time_us":96211}
I20250113 08:59:11.187213 14497 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:11.221194 11887 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:11.256763 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.243964 11888 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:11.193560 14406 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:11.229686 14542 log.cc:926] Time spent T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20250113 08:59:11.240816 11963 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:11.201313 14548 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:11.201738 14194 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.228965 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:11.222954 14271 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.222492 14441 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:11.258949 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 176323us
I20250113 08:59:11.257874 11960 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
W20250113 08:59:11.257663 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 110357us
I20250113 08:59:11.259707 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:11.316509 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 111130us
W20250113 08:59:11.251175 12036 consensus_meta.cc:287] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.974s user 0.001s sys 0.011s
W20250113 08:59:11.314978 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 109836us
I20250113 08:59:11.223119 14528 consensus_queue.cc:1046] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
I20250113 08:59:11.202951 14557 consensus_queue.cc:1035] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.143s
W20250113 08:59:11.339298 14546 log.cc:926] Time spent T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.221s user 0.000s sys 0.000s
I20250113 08:59:11.258608 11967 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250113 08:59:11.340786 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.369204 11885 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:11.341572 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:11.237636 14421 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:11.190564 14518 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:11.257504 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.224596 14391 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:11.349725 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 94800.2us
I20250113 08:59:11.226200 14488 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
I20250113 08:59:11.202507 14358 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.326011 14525 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:11.229130 14477 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.365702 11856 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:11.370604 12723 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:11.259614 14438 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:11.336963 14548 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:11.378299 11859 leader_election.cc:336] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 2.086s (SENT)
I20250113 08:59:11.228672 14522 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.429481 11966 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:11.429577 14522 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:11.427126 14438 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.379715 14551 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:11.431257 14551 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:11.432189 14551 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:11.380156 14547 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:11.433288 14461 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:11.433465 14547 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:11.433657 14212 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.434003 14547 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:11.434299 14212 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.434731 14212 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.436676 14212 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.437589 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2187) took 1370 ms. Trace:
I20250113 08:59:11.380520 14333 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.384121 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2142) took 3233 ms. Trace:
I20250113 08:59:11.438117 12042 rpcz_store.cc:276] 0113 08:59:10.066840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.375848 (+309008us) service_pool.cc:225] Handling call
0113 08:59:10.513455 (+137607us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:10.553165 (+ 39710us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:10.553210 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:10.561331 (+ 8121us) log.cc:844] Serialized 26 byte log entry
0113 08:59:10.753561 (+192230us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:10.969375 (+215814us) log.cc:844] Serialized 14 byte log entry
0113 08:59:11.101737 (+132362us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.101851 (+ 114us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:11.437312 (+335461us) raft_consensus.cc:1680] finished
0113 08:59:11.437338 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:11.437502 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6367,"prepare.queue_time_us":571804,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":122004,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":20081,"thread_start_us":452483,"threads_started":3,"wal-append.queue_time_us":354457}
I20250113 08:59:11.384346 12720 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:11.441555 11934 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:11.442879 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:11.443818 14344 consensus_meta.cc:287] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.515s user 0.007s sys 0.000s
I20250113 08:59:11.444242 14344 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:11.445283 11964 raft_consensus.cc:1235] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 2. Current term is 3. Ops: []
I20250113 08:59:11.445147 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:11.446111 14333 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:11.446833 14333 raft_consensus.cc:695] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:11.447728 14333 consensus_queue.cc:237] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.441068 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:11.454246 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:11.455412 12042 raft_consensus.cc:2388] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:11.455276 14539 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.456552 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:11.389747 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.350641 14406 raft_consensus.cc:695] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:11.202324 14392 consensus_queue.cc:260] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.201013 14543 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.362447 14537 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:11.353194 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.421841 12040 raft_consensus.cc:2388] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:11.512657 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.514858 14543 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:11.461751 14552 consensus_queue.cc:1159] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" responder_term: 5 status { last_received { term: 3 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.7-4.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20250113 08:59:11.518569 14481 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:11.519060 14481 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:11.416038 14473 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:11.423305 14525 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250113 08:59:11.424063 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 58482us
W20250113 08:59:11.424618 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 46344.7us
I20250113 08:59:11.345886 14194 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.425688 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:11.425746 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.535313 11885 raft_consensus.cc:2405] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:11.259263 14533 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.411495 14528 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20250113 08:59:11.225498 14553 consensus_queue.cc:1046] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.201s
I20250113 08:59:11.364543 14271 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20250113 08:59:11.360013 12036 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:11.428769 14358 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.430473 14518 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:11.440591 14409 consensus_meta.cc:287] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.520s user 0.003s sys 0.007s
I20250113 08:59:11.353237 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.339588 14555 consensus_queue.cc:1046] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.200s
W20250113 08:59:11.459750 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 105685us
I20250113 08:59:11.378693 14562 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:11.353901 12717 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:11.394969 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:11.440169 11962 rpcz_store.cc:276] 0113 08:59:08.150381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.579288 (+2428907us) service_pool.cc:225] Handling call
0113 08:59:10.598531 (+ 19243us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.244535 (+646004us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.383890 (+139355us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540006e5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:11.384062 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1396272896}
I20250113 08:59:11.503805 11891 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:11.501801 12042 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:11.505208 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:11.507758 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.375167 14123 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:11.522161 11886 raft_consensus.cc:2405] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:11.522948 11892 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20250113 08:59:11.609467 11892 pending_rounds.cc:77] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 5
I20250113 08:59:11.609769 11892 pending_rounds.cc:99] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20250113 08:59:11.505089 14291 consensus_meta.cc:287] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.205s user 0.008s sys 0.005s
I20250113 08:59:11.422405 14477 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.610055 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.611133 11962 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:11.611263 14150 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.612155 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.612001 14150 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:11.390506 14441 consensus_queue.cc:1046] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.134s
I20250113 08:59:11.610050 11892 raft_consensus.cc:2883] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:11.593470 14409 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:11.539268 11856 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:11.509912 14539 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:11.576576 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.416080 14560 consensus_queue.cc:1046] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.204s
I20250113 08:59:11.589553 11887 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:11.532791 11959 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:11.591107 12686 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:11.506619 14547 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:11.593693 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 368701us
I20250113 08:59:11.641031 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.641460 12006 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:11.590711 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.590276 11963 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:11.643139 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.643592 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.644452 12724 raft_consensus.cc:1235] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:11.225955 14558 consensus_queue.cc:1046] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.262s
I20250113 08:59:11.589133 14525 raft_consensus.cc:3049] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
W20250113 08:59:11.622748 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 83587.8us
I20250113 08:59:11.615705 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:11.537433 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:11.628135 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 271762us
W20250113 08:59:11.638075 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 137124us
I20250113 08:59:11.641460 11934 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:11.615239 11967 raft_consensus.cc:2405] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250113 08:59:11.585932 14570 raft_consensus.cc:2743] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:11.545871 14242 consensus_queue.cc:1046] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20250113 08:59:11.646206 11963 raft_consensus.cc:2388] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:11.545799 14543 consensus_queue.cc:1046] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20250113 08:59:11.615741 14358 raft_consensus.cc:3049] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:11.646137 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:11.609033 12036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 500) took 2439 ms (client timeout 2143 ms). Trace:
I20250113 08:59:11.654897 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.646781 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.614821 14406 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:11.884649 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 241931us
I20250113 08:59:11.886061 11858 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:11.670324 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:11.888131 11889 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 3.
I20250113 08:59:11.888938 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:11.889811 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 298821us
I20250113 08:59:11.887171 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:11.661651 11962 raft_consensus.cc:2405] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250113 08:59:11.661237 11856 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:11.769729 12035 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:11.895991 11960 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:11.619403 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 698) took 2879 ms. Trace:
I20250113 08:59:11.769276 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2182) took 1976 ms. Trace:
W20250113 08:59:11.767815 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2192) took 1613 ms (client timeout 1573 ms). Trace:
W20250113 08:59:11.898088 11888 consensus_meta.cc:287] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.559s user 0.010s sys 0.000s
W20250113 08:59:11.902035 12040 rpcz_store.cc:269] 0113 08:59:10.154730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.187099 (+ 32369us) service_pool.cc:225] Handling call
0113 08:59:11.767471 (+1580372us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540024ead4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:11.767708 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2771411840}
I20250113 08:59:11.902390 11888 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:11.901890 12037 rpcz_store.cc:276] 0113 08:59:09.793302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.793493 (+ 191us) service_pool.cc:225] Handling call
0113 08:59:09.793915 (+ 422us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.254582 (+1460667us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.768918 (+514336us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b540021dc54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:11.769168 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3282891136}
I20250113 08:59:11.903546 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.662329 12720 raft_consensus.cc:2405] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:11.906353 12006 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:11.912868 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.913777 11885 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:11.915760 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 254608us
I20250113 08:59:11.916829 11856 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:11.669152 14342 consensus_meta.cc:287] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.697s user 0.008s sys 0.007s
I20250113 08:59:11.908174 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:11.921342 12723 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250113 08:59:11.918673 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.643492 14562 raft_consensus.cc:695] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:11.919816 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:11.889636 14438 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.945845 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.918744 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:11.669345 14461 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:11.889407 14553 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.814790 14518 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.732714 14551 raft_consensus.cc:3049] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:11.770426 14497 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:11.945101 12037 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:11.742377 14123 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:11.923961 12686 leader_election.cc:400] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4.
I20250113 08:59:11.836820 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:11.866523 14344 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:11.661160 14477 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:11.951906 11962 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:11.959570 14579 log.cc:926] Time spent T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20250113 08:59:11.905747 14291 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:11.952937 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 46709.4us
I20250113 08:59:11.953378 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:11.953981 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 566) took 3112 ms. Trace:
I20250113 08:59:12.010700 12723 rpcz_store.cc:276] 0113 08:59:08.841934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.908414 (+3066480us) service_pool.cc:225] Handling call
0113 08:59:11.908813 (+ 399us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.953717 (+ 44904us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.953923 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:11.954661 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:12.012115 11960 raft_consensus.cc:2388] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 3: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:11.967770 14242 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:11.864424 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:11.660436 14525 raft_consensus.cc:738] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:11.954535 14461 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:11.897472 14358 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:11.959533 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:12.021690 14358 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.022557 11889 raft_consensus.cc:2388] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:12.022938 14358 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:12.023236 14576 log.cc:926] Time spent T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20250113 08:59:12.023612 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 512) took 1734 ms. Trace:
W20250113 08:59:12.024147 14571 log.cc:926] Time spent T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.400s user 0.001s sys 0.000s
I20250113 08:59:12.023977 12041 rpcz_store.cc:276] 0113 08:59:10.288725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.581018 (+292293us) service_pool.cc:225] Handling call
0113 08:59:11.582153 (+1001135us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.582553 (+ 400us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:11.582593 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:11.582653 (+ 60us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:11.582705 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.582773 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:12.023426 (+440653us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b540021d750. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:12.023535 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8842,"raft.queue_time_us":528,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7596,"spinlock_wait_cycles":2248500992}
I20250113 08:59:11.959877 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:11.960328 11888 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250113 08:59:11.961740 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 704) took 2970 ms (client timeout 1724 ms). Trace:
I20250113 08:59:12.019189 11959 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:11.961824 12726 raft_consensus.cc:1235] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:12.026099 12725 rpcz_store.cc:269] 0113 08:59:08.991799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.100326 (+108527us) service_pool.cc:225] Handling call
0113 08:59:11.961500 (+2861174us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b5400751554. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:11.961657 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4526776192}
I20250113 08:59:12.026744 11888 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:11.901860 12718 rpcz_store.cc:276] 0113 08:59:08.739721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.200124 (+2460403us) service_pool.cc:225] Handling call
0113 08:59:11.200526 (+ 402us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.252247 (+ 51721us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:11.359335 (+107088us) log.cc:844] Serialized 14 byte log entry
0113 08:59:11.588189 (+228854us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:11.592188 (+ 3999us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:11.606240 (+ 14052us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.614599 (+ 8359us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:11.614751 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7317,"prepare.queue_time_us":88232,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":6780,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":15786,"thread_start_us":171977,"threads_started":1,"wal-append.queue_time_us":359186}
I20250113 08:59:12.027364 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 559) took 4715 ms. Trace:
I20250113 08:59:12.027513 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:11.998850 12717 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 2 FOLLOWER]: Advancing to term 4
I20250113 08:59:11.993485 12686 leader_election.cc:403] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:11.973675 11931 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:11.952680 14488 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:11.991427 14497 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:11.972230 11885 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
W20250113 08:59:11.960901 14572 log.cc:926] Time spent T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
W20250113 08:59:11.954666 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 54508.8us
I20250113 08:59:11.956465 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:12.014750 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1636) took 1421 ms. Trace:
I20250113 08:59:12.019841 14525 consensus_queue.cc:260] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:12.020676 14461 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:11.959704 14562 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.973358 14438 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.028215 12725 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 6
I20250113 08:59:12.048607 12006 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:11.923664 14552 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:12.025671 12033 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:12.027737 12726 rpcz_store.cc:276] 0113 08:59:07.312162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.924563 (+3612401us) service_pool.cc:225] Handling call
0113 08:59:11.961628 (+1037065us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.026617 (+ 64989us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.027203 (+ 586us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400751550. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:12.027310 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2377175424}
I20250113 08:59:12.031947 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:12.032325 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 58792.6us
I20250113 08:59:12.033139 11962 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:12.033139 12037 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:11.965978 14551 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:12.058982 14551 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:12.060177 14551 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:11.959020 14441 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:12.061007 14473 raft_consensus.cc:3049] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:12.061465 14441 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.061902 14441 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:11.993760 14407 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20250113 08:59:11.994326 14476 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:11.765329 14129 raft_consensus.cc:3049] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:12.064365 14129 raft_consensus.cc:738] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:11.958202 14575 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.119s
I20250113 08:59:12.065151 14129 consensus_queue.cc:260] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:12.047359 14580 consensus_queue.cc:1046] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20250113 08:59:12.070264 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 511) took 1222 ms. Trace:
I20250113 08:59:12.070394 14547 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:12.071084 14549 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4.
I20250113 08:59:12.071863 14525 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:11.897557 12036 rpcz_store.cc:269] 0113 08:59:09.148627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.152354 (+ 3727us) service_pool.cc:225] Handling call
0113 08:59:11.588394 (+2436040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.044389 12040 rpcz_store.cc:276] 0113 08:59:10.592941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.909654 (+1316713us) service_pool.cc:225] Handling call
0113 08:59:12.014686 (+105032us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.073292 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:12.073743 14421 consensus_meta.cc:287] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.680s user 0.008s sys 0.001s
I20250113 08:59:12.074160 14421 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.074532 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.075664 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.075908 14541 raft_consensus.cc:3049] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
I20250113 08:59:12.044696 11967 raft_consensus.cc:2405] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:12.076344 14541 raft_consensus.cc:738] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:12.005482 14477 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.077385 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.044754 12007 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.079188 11966 raft_consensus.cc:1235] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250113 08:59:12.016171 14242 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.080595 12007 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.012383 14458 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:12.027789 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:11.960414 14553 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:12.055498 11931 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.055876 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1652) took 1166 ms. Trace:
W20250113 08:59:12.057652 14211 consensus_meta.cc:287] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.958s user 0.002s sys 0.009s
I20250113 08:59:12.057898 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.058089 14573 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:12.069500 12006 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:11.950078 14342 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.995817 14533 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:12.033191 14333 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.021s sys 0.021s
W20250113 08:59:12.101977 14333 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.021s sys 0.022s
I20250113 08:59:12.102782 14333 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.103253 14333 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:11.974753 14518 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.061487 14473 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:12.076381 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1637) took 1483 ms. Trace:
I20250113 08:59:12.080509 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:12.111158 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 469742us
I20250113 08:59:12.010815 14123 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.070688 11890 rpcz_store.cc:276] 0113 08:59:10.848330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.848485 (+ 155us) service_pool.cc:225] Handling call
0113 08:59:10.875178 (+ 26693us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.509900 (+634722us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:11.509958 (+ 58us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:11.510041 (+ 83us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:11.510104 (+ 63us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.070036 (+559932us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b5400090454. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:12.070084 (+ 48us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:12.070201 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":127,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":4427,"spinlock_wait_cycles":1449848576}
I20250113 08:59:11.966688 14581 consensus_queue.cc:1046] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.103s
W20250113 08:59:12.071869 14351 consensus_meta.cc:287] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.820s user 0.010s sys 0.000s
I20250113 08:59:12.101092 14555 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:12.150238 11893 raft_consensus.cc:1270] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250113 08:59:12.077430 14541 consensus_queue.cc:260] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:12.119766 12037 rpcz_store.cc:276] 0113 08:59:10.889213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.903999 (+1014786us) service_pool.cc:225] Handling call
0113 08:59:12.055794 (+151795us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.120680 12040 rpcz_store.cc:276] 0113 08:59:10.593217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.075450 (+1482233us) service_pool.cc:225] Handling call
0113 08:59:12.076331 (+ 881us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.163161 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:12.164047 12040 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:12.086545 12007 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:12.119931 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 217538us
I20250113 08:59:12.079914 14344 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:12.077468 14477 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:12.098752 11931 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:12.123395 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2196) took 1192 ms. Trace:
I20250113 08:59:12.167078 12041 rpcz_store.cc:276] 0113 08:59:10.931287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.040443 (+1109156us) service_pool.cc:225] Handling call
0113 08:59:12.123332 (+ 82889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.168339 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:12.124468 14333 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:12.078622 14552 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:12.170166 14552 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:12.076304 14421 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:12.122521 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 205819us
I20250113 08:59:12.090147 14242 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:12.082851 14575 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:12.091922 14458 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.174098 14458 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:12.174479 14575 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.174989 14575 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.177508 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:12.179603 12726 raft_consensus.cc:1235] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:12.180644 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 707) took 2176 ms. Trace:
I20250113 08:59:12.180744 14528 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader election won for term 9
I20250113 08:59:12.181519 14528 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:12.181676 14333 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:12.182137 14333 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:12.182521 14333 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:12.182621 14528 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.124323 14211 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.205327 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.212780 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.213769 12037 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:12.158562 14555 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.220803 14555 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:12.081799 14438 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:12.154835 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1654) took 1003 ms. Trace:
I20250113 08:59:12.181149 12726 rpcz_store.cc:276] 0113 08:59:10.003972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.125030 (+2121058us) service_pool.cc:225] Handling call
0113 08:59:12.147964 (+ 22934us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.180306 (+ 32342us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.180577 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.215126 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1647) took 1606 ms. Trace:
W20250113 08:59:12.218871 14584 log.cc:926] Time spent T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20250113 08:59:12.198947 14267 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:12.231570 12722 raft_consensus.cc:1235] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:12.243888 12037 rpcz_store.cc:276] 0113 08:59:10.608630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.212516 (+1603886us) service_pool.cc:225] Handling call
0113 08:59:12.215059 (+ 2543us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.206493 14549 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.257414 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:12.262741 11962 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:12.261384 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:12.148741 14351 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.264139 12726 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:12.265040 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:12.266034 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 48026.3us
W20250113 08:59:12.266528 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 345754us
I20250113 08:59:12.265533 11934 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:12.265414 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 166793us
I20250113 08:59:12.268767 11931 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:12.267571 12686 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.257663 14541 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:12.275583 11856 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:12.275466 11960 raft_consensus.cc:1235] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:12.278617 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:12.279523 14351 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:12.280093 14150 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:12.282917 12717 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:12.284893 14575 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:12.285794 11959 raft_consensus.cc:1235] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:12.286767 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 505) took 2079 ms (client timeout 2180 ms). Trace:
W20250113 08:59:12.287145 12039 rpcz_store.cc:269] 0113 08:59:10.207839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.371020 (+163181us) service_pool.cc:225] Handling call
0113 08:59:12.286574 (+1915554us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b540020cbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:12.286694 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1484733696}
I20250113 08:59:12.222297 14344 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250113 08:59:12.288154 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:12.288873 12039 raft_consensus.cc:2388] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:12.289882 14553 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:12.290251 14461 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.292277 14461 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:12.293843 14438 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:12.232018 12036 rpcz_store.cc:276] 0113 08:59:11.134517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.077078 (+942561us) service_pool.cc:225] Handling call
0113 08:59:12.137477 (+ 60399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.294466 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:12.294055 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:12.305393 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:12.306702 11856 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:12.301815 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.308429 11961 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:12.320195 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2156) took 1064 ms. Trace:
W20250113 08:59:12.309870 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 44421.7us
I20250113 08:59:12.311511 11931 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.320508 11961 rpcz_store.cc:276] 0113 08:59:11.255650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.293756 (+1038106us) service_pool.cc:225] Handling call
0113 08:59:12.320132 (+ 26376us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:12.313808 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 46318.1us
I20250113 08:59:12.295028 14391 raft_consensus.cc:2743] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:12.321992 12686 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:12.319306 12033 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:12.323995 11934 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:12.325274 12039 raft_consensus.cc:2363] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6.
I20250113 08:59:12.329802 12040 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:12.336359 12040 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:12.266336 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:12.338331 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 251884us
I20250113 08:59:12.338505 12684 leader_election.cc:400] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6.
I20250113 08:59:12.339468 12684 leader_election.cc:403] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:12.340734 12684 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:12.298363 11891 consensus_meta.cc:287] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.678s user 0.017s sys 0.001s
I20250113 08:59:12.346068 11891 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:12.346976 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1776) took 1128 ms. Trace:
I20250113 08:59:12.338789 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.347455 11891 rpcz_store.cc:276] 0113 08:59:11.218951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.363073 (+144122us) service_pool.cc:225] Handling call
0113 08:59:12.346899 (+983826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.351867 12725 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:12.197204 14473 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:59:12.355420 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 44006us
I20250113 08:59:12.356181 11931 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.357195 11931 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:12.357823 11858 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:12.358099 11891 raft_consensus.cc:1235] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:12.358628 12037 raft_consensus.cc:2388] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:12.358691 12007 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.353096 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:12.315498 14537 consensus_meta.cc:287] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.796s user 0.015s sys 0.005s
I20250113 08:59:12.337517 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 516) took 2047 ms. Trace:
I20250113 08:59:12.365828 12040 rpcz_store.cc:276] 0113 08:59:10.290076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.335574 (+2045498us) service_pool.cc:225] Handling call
0113 08:59:12.335904 (+ 330us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.337216 (+ 1312us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.337458 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.366951 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1639) took 1773 ms. Trace:
I20250113 08:59:12.366983 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.367516 12037 rpcz_store.cc:276] 0113 08:59:10.593737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.338543 (+1744806us) service_pool.cc:225] Handling call
0113 08:59:12.366888 (+ 28345us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.367928 12040 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:12.369369 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:12.363565 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 195972us
I20250113 08:59:12.370052 12040 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:12.371460 12008 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.303987 14589 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:12.385993 11858 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:12.386327 11931 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:12.267652 14549 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.338239 14539 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:12.389952 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.402046 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.402776 12040 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:12.403811 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.382766 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:12.390913 11858 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:12.359385 14455 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:12.376277 14543 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.358280 14351 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:12.392210 12008 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.338618 14150 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:12.392311 12007 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.393561 11931 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.339395 14580 raft_consensus.cc:2743] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:12.354066 14344 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:12.399600 11891 pending_rounds.cc:77] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 5
I20250113 08:59:12.442027 11891 pending_rounds.cc:99] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250113 08:59:12.442389 11891 raft_consensus.cc:2883] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:12.391145 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:12.443531 12037 raft_consensus.cc:2388] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:12.409668 11856 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:12.297979 14587 consensus_queue.cc:1046] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.126s
I20250113 08:59:12.354631 14460 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231) with higher term. Message: Invalid argument: T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6.
I20250113 08:59:12.385200 14529 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:12.446375 14529 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:12.446878 14529 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:12.381904 14267 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:12.447964 11856 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:12.449874 14553 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:12.450428 14553 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:12.450902 14553 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:12.412102 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2152) took 1173 ms. Trace:
I20250113 08:59:12.453773 11968 rpcz_store.cc:276] 0113 08:59:11.239152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.239272 (+ 120us) service_pool.cc:225] Handling call
0113 08:59:11.254282 (+ 15010us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:11.314818 (+ 60536us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:11.314862 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:11.315441 (+ 579us) log.cc:844] Serialized 26 byte log entry
0113 08:59:11.775564 (+460123us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b18002dc0a0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336b2e55 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336d08c2 00007fc336285e4b 00007fc33628b913 00007fc33628bcc1 00007fc3365dd45f
0113 08:59:11.876967 (+101403us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:59:11.877051 (+ 84us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:11.897973 (+ 20922us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:12.400977 (+503004us) raft_consensus.cc:1680] finished
0113 08:59:12.400993 (+ 16us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:12.412000 (+ 11007us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1326,"raft.queue_time_us":3144,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":8331,"spinlock_wait_cycles":10304384,"thread_start_us":65416,"threads_started":1,"wal-append.queue_time_us":692981}
I20250113 08:59:12.456382 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:12.457135 11968 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
W20250113 08:59:12.463943 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 71957.9us
I20250113 08:59:12.422037 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.426208 11931 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.394810 14588 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:12.359529 14533 consensus_queue.cc:1046] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20250113 08:59:12.408635 12718 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:12.467617 11856 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:12.468950 12040 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250113 08:59:12.450903 14562 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250113 08:59:12.469777 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 77516.6us
I20250113 08:59:12.398494 14547 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:12.385022 14407 consensus_queue.cc:1046] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20250113 08:59:12.412053 14123 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:12.429868 14444 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.493s user 0.007s sys 0.023s
I20250113 08:59:12.409727 14473 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:12.409647 14518 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:12.409514 14342 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:12.482859 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1634) took 1890 ms. Trace:
I20250113 08:59:12.481163 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:12.479951 12008 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:12.482200 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 56061.1us
I20250113 08:59:12.482237 12037 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250113 08:59:12.493930 12007 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.494470 14581 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:12.495015 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 703) took 3651 ms. Trace:
I20250113 08:59:12.393652 14591 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:12.508322 14461 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:12.512670 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45172us
I20250113 08:59:12.392136 14558 consensus_queue.cc:1046] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20250113 08:59:12.391939 14421 consensus_queue.cc:1035] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20250113 08:59:12.424717 14351 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.427251 14476 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:12.509582 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 187270us
W20250113 08:59:12.473779 14548 consensus_meta.cc:287] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.018s user 0.008s sys 0.014s
I20250113 08:59:12.429848 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.499754 14562 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:12.549844 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:12.471560 14455 raft_consensus.cc:695] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:12.503469 14547 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.548192 11960 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:12.413290 14573 raft_consensus.cc:695] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:12.547936 12040 rpcz_store.cc:276] 0113 08:59:10.592363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.468295 (+1875932us) service_pool.cc:225] Handling call
0113 08:59:12.468585 (+ 290us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.482383 (+ 13798us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.482805 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.593487 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1649) took 1892 ms. Trace:
I20250113 08:59:12.616488 12037 rpcz_store.cc:276] 0113 08:59:10.700901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.470845 (+1769944us) service_pool.cc:225] Handling call
0113 08:59:12.471149 (+ 304us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.575394 (+104245us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.593421 (+ 18027us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:12.523711 14344 raft_consensus.cc:3049] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:12.594707 11931 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:12.522496 14444 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.602s user 0.007s sys 0.023s
I20250113 08:59:12.429739 14352 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20250113 08:59:12.604218 11934 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:12.607079 11968 raft_consensus.cc:2405] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250113 08:59:12.608479 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 128630us
I20250113 08:59:12.434311 14543 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.523455 14588 raft_consensus.cc:695] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:12.523871 14211 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:12.618438 11931 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.478775 14438 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:12.527302 14407 consensus_queue.cc:1225] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:12.478286 14212 consensus_queue.cc:1046] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.112s
W20250113 08:59:12.608865 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 115041us
I20250113 08:59:12.528467 12717 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:12.625118 11884 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:12.649747 14518 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:12.441044 14589 consensus_queue.cc:1046] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.177s
I20250113 08:59:12.629415 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:12.527209 14141 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:12.575866 14488 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:12.596748 14591 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.602393 14351 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:12.597754 14601 raft_consensus.cc:2743] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:12.623133 14344 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:12.606046 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1645) took 2010 ms. Trace:
W20250113 08:59:12.644698 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 180684us
I20250113 08:59:12.572284 14604 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:12.531100 14587 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:12.622277 14342 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.606267 14562 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:12.602646 14476 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:59:12.643325 14525 consensus_meta.cc:287] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.567s user 0.009s sys 0.004s
I20250113 08:59:12.823015 14553 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.714428 12007 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:12.607637 14547 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:12.825960 14619 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:59:12.665978 14539 raft_consensus.cc:695] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:12.630507 14123 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
W20250113 08:59:12.657357 14497 consensus_meta.cc:287] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.558s user 0.010s sys 0.003s
I20250113 08:59:12.628115 14558 raft_consensus.cc:3049] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:12.648715 14585 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:12.726861 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:12.604558 14548 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:12.717778 14481 consensus_meta.cc:287] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.198s user 0.009s sys 0.006s
I20250113 08:59:12.596036 14581 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:12.715080 14574 log.cc:926] Time spent T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250113 08:59:12.642892 14613 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:12.695784 14407 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.737219 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:12.718659 14212 consensus_queue.cc:1225] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:12.607412 14455 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:12.748212 14592 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20250113 08:59:12.742062 14591 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:12.749302 14599 log.cc:926] Time spent T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20250113 08:59:12.643427 14333 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.732846 14391 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:12.735379 14141 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:12.738569 14444 consensus_queue.cc:1046] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20250113 08:59:12.768379 11960 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:12.771380 14612 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:12.772198 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2159) took 1418 ms. Trace:
I20250113 08:59:12.772529 11884 raft_consensus.cc:1270] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
W20250113 08:59:12.774775 14441 consensus_meta.cc:287] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.710s user 0.023s sys 0.000s
I20250113 08:59:12.750909 14589 consensus_queue.cc:1225] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20250113 08:59:12.757436 14596 log.cc:926] Time spent T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20250113 08:59:12.815335 14342 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:12.816131 14605 log.cc:926] Time spent T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20250113 08:59:12.811921 14587 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:59:12.816437 11934 leader_election.cc:336] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 2.127s (SENT)
W20250113 08:59:12.817714 14552 consensus_meta.cc:287] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.647s user 0.016s sys 0.004s
W20250113 08:59:12.818331 14541 consensus_meta.cc:287] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.537s user 0.008s sys 0.001s
W20250113 08:59:12.826217 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 111902us
I20250113 08:59:13.152364 11884 pending_rounds.cc:77] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 4
W20250113 08:59:13.059979 11964 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.006s sys 0.010s
W20250113 08:59:13.152555 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 336143us
I20250113 08:59:12.819862 14608 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:59:12.780385 14601 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:12.821378 14476 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:13.154882 12040 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:13.151657 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 235486us
I20250113 08:59:13.153811 11965 rpcz_store.cc:276] 0113 08:59:11.354115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.994765 (+640650us) service_pool.cc:225] Handling call
0113 08:59:12.016203 (+ 21438us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.045431 (+ 29228us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.045477 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.054140 (+ 8663us) log.cc:844] Serialized 26 byte log entry
0113 08:59:12.401790 (+347650us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:12.469154 (+ 67364us) log.cc:844] Serialized 14 byte log entry
0113 08:59:12.638878 (+169724us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.641053 (+ 2175us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:12.771937 (+130884us) raft_consensus.cc:1680] finished
0113 08:59:12.771965 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:12.772126 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39733,"prepare.queue_time_us":377062,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":3155,"raft.queue_time_us":118773,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":45172,"thread_start_us":523258,"threads_started":3,"wal-append.queue_time_us":668583}
W20250113 08:59:13.150170 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 396287us
W20250113 08:59:13.151311 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.010s sys 0.014s
W20250113 08:59:13.170039 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.010s sys 0.014s
I20250113 08:59:13.170233 11858 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:13.152706 11884 pending_rounds.cc:99] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250113 08:59:13.171061 11884 raft_consensus.cc:2883] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:12.758658 14549 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:13.153198 11964 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s user 0.009s sys 0.011s
I20250113 08:59:13.157634 11891 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:13.152613 12007 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:12.960034 14555 consensus_meta.cc:287] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.704s user 0.008s sys 0.003s
I20250113 08:59:13.162050 14141 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:13.181340 14297 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250113 08:59:12.778216 14344 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.163450 14497 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.167471 14352 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:12.830982 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 514) took 2541 ms. Trace:
I20250113 08:59:13.173956 11960 raft_consensus.cc:1235] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:13.174188 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:13.156765 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 516) took 1793 ms. Trace:
I20250113 08:59:13.178452 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1759) took 4574 ms. Trace:
I20250113 08:59:13.214960 11884 rpcz_store.cc:276] 0113 08:59:08.597552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:08.776103 (+178551us) service_pool.cc:225] Handling call
0113 08:59:08.843073 (+ 66970us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:09.752578 (+909505us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b5400360220. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336599990 00007fc33660b300 00007fc33660c371 00007fc336608edf 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc
0113 08:59:10.299532 (+546954us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b5400360220. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33659766d 00007fc3365852b0 00007fc3365e8aba 00007fc3366003af 00007fc336601b4e 00007fc33660bc89 00007fc33660c556 00007fc336608edf 00007fc33660899e 00007fc33b5672d9
0113 08:59:12.608603 (+2309071us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b580062eab0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc3365851c3 00007fc3365e8aba 00007fc3366003af 00007fc336601b4e 00007fc33660bc89 00007fc33660c556 00007fc336608edf 00007fc33660899e 00007fc33b5672d9 00007fc334c33694
0113 08:59:13.171503 (+562900us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.171708 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3049508736}
I20250113 08:59:13.178192 14438 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:13.161463 14558 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:13.229454 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 514) took 1877 ms. Trace:
W20250113 08:59:13.038012 14477 consensus_meta.cc:287] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.830s user 0.010s sys 0.001s
I20250113 08:59:13.230021 11888 rpcz_store.cc:276] 0113 08:59:11.351820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.027513 (+675693us) service_pool.cc:225] Handling call
0113 08:59:12.027804 (+ 291us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.028140 (+ 336us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.028180 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.080394 (+ 52214us) log.cc:844] Serialized 26 byte log entry
0113 08:59:12.398551 (+318157us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:12.398587 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.398654 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:13.229190 (+830536us) raft_consensus.cc:1680] finished
0113 08:59:13.229211 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:13.229383 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1505,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":12033,"thread_start_us":281872,"threads_started":1,"wal-append.queue_time_us":419545}
W20250113 08:59:13.185516 11963 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.037s user 0.018s sys 0.016s
W20250113 08:59:13.232220 11963 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.083s user 0.019s sys 0.016s
I20250113 08:59:13.186861 12041 raft_consensus.cc:2388] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:13.191459 11961 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.018s sys 0.022s
W20250113 08:59:13.233271 11961 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.018s sys 0.023s
W20250113 08:59:13.035830 14406 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.836s user 0.005s sys 0.017s
I20250113 08:59:12.752178 12036 rpcz_store.cc:276] 0113 08:59:10.595441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.359690 (+1764249us) service_pool.cc:225] Handling call
0113 08:59:12.605344 (+245654us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:13.008656 14609 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:12.758005 14573 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:12.995891 14518 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:12.949182 14604 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:12.754875 14588 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.130693 14543 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:13.206388 11965 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:13.168651 14587 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:13.210796 11967 raft_consensus.cc:1270] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250113 08:59:13.267288 11967 pending_rounds.cc:77] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting all ops after (but not including) 1
I20250113 08:59:13.267643 11967 pending_rounds.cc:99] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250113 08:59:13.267947 11967 raft_consensus.cc:2883] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:12.960608 14461 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:13.179665 14541 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:13.219801 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 515) took 1867 ms. Trace:
I20250113 08:59:13.213376 12040 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:13.279371 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:13.280030 12040 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:13.213743 11887 rpcz_store.cc:276] 0113 08:59:11.363080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.062404 (+699324us) service_pool.cc:225] Handling call
0113 08:59:12.062731 (+ 327us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.063030 (+ 299us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.063057 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.063479 (+ 422us) log.cc:844] Serialized 26 byte log entry
0113 08:59:12.413040 (+349561us) raft_consensus.cc:1640] Marking committed up to 8
0113 08:59:12.413099 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.413196 (+ 97us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:12.751028 (+337832us) raft_consensus.cc:1680] finished
0113 08:59:12.751047 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:13.156693 (+405646us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1492,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":52,"thread_start_us":349184,"threads_started":1,"wal-append.queue_time_us":631995}
I20250113 08:59:13.242869 14344 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:12.829156 14593 log.cc:926] Time spent T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20250113 08:59:13.212618 12033 rpcz_store.cc:276] 0113 08:59:10.289420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.331092 (+2041672us) service_pool.cc:225] Handling call
0113 08:59:12.331526 (+ 434us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.357703 (+ 26177us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.357729 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.394266 (+ 36537us) log.cc:844] Serialized 26 byte log entry
0113 08:59:12.643566 (+249300us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:12.643613 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.643681 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:12.830650 (+186969us) raft_consensus.cc:1680] finished
0113 08:59:12.830679 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:12.830898 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21783,"raft.queue_time_us":111657,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":50061,"thread_start_us":168840,"threads_started":1,"wal-append.queue_time_us":361891}
I20250113 08:59:13.179143 14553 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:13.222536 14581 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:13.227530 14608 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:13.227841 14333 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:13.228924 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1911) took 2006 ms. Trace:
I20250113 08:59:13.205722 14527 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:13.238906 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:13.239502 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 86964.5us
I20250113 08:59:13.237393 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 520) took 1523 ms. Trace:
I20250113 08:59:13.240339 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:13.169202 14481 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.003559 14562 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:13.330960 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:13.172070 14552 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.173363 14441 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.173290 14407 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.207113 14555 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:13.237496 14406 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.038s user 0.006s sys 0.024s
I20250113 08:59:13.257550 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:13.209934 14549 consensus_queue.cc:1046] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20250113 08:59:13.227909 14548 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:13.245334 14529 consensus_meta.cc:287] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.793s user 0.003s sys 0.010s
W20250113 08:59:13.336633 14358 consensus_meta.cc:287] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.313s user 0.024s sys 0.000s
I20250113 08:59:13.283124 11856 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:13.360206 12039 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:13.216991 11884 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:13.277735 11885 rpcz_store.cc:276] 0113 08:59:11.352165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.041156 (+688991us) service_pool.cc:225] Handling call
0113 08:59:12.041547 (+ 391us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:12.056607 (+ 15060us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.056639 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:12.057168 (+ 529us) log.cc:844] Serialized 52 byte log entry
0113 08:59:12.384878 (+327710us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:12.384923 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.385122 (+ 199us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:13.219593 (+834471us) raft_consensus.cc:1680] finished
0113 08:59:13.219613 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:13.219735 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15898,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":327283,"threads_started":1,"wal-append.queue_time_us":465046}
I20250113 08:59:13.222816 14352 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.235826 14267 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:13.297319 11889 rpcz_store.cc:276] 0113 08:59:11.222172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.024255 (+802083us) service_pool.cc:225] Handling call
0113 08:59:12.024560 (+ 305us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.024916 (+ 356us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.024938 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.039961 (+ 15023us) log.cc:844] Serialized 26 byte log entry
0113 08:59:12.362038 (+322077us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:59:12.362088 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.362172 (+ 84us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:13.228697 (+866525us) raft_consensus.cc:1680] finished
0113 08:59:13.228723 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:13.228856 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":263,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":280383,"threads_started":1,"wal-append.queue_time_us":429067}
I20250113 08:59:13.301507 11890 rpcz_store.cc:276] 0113 08:59:11.714472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.230979 (+516507us) service_pool.cc:225] Handling call
0113 08:59:12.231381 (+ 402us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.231605 (+ 224us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.231634 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.243477 (+ 11843us) log.cc:844] Serialized 26 byte log entry
0113 08:59:12.501023 (+257546us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:12.501072 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.501167 (+ 95us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:13.237113 (+735946us) raft_consensus.cc:1680] finished
0113 08:59:13.237139 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:13.237318 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":227130,"threads_started":1,"wal-append.queue_time_us":338807}
I20250113 08:59:13.328132 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:13.326669 11888 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:13.362928 11967 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:13.264668 14601 raft_consensus.cc:695] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:13.251077 14391 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:13.331148 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:13.332376 12007 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:13.236948 14477 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.364796 12033 raft_consensus.cc:1235] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250113 08:59:13.297199 14629 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:13.371922 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:13.372766 11885 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:13.373099 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2195) took 2458 ms. Trace:
I20250113 08:59:13.373502 12033 rpcz_store.cc:276] 0113 08:59:10.914814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.334927 (+2420113us) service_pool.cc:225] Handling call
0113 08:59:13.360025 (+ 25098us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:13.372871 (+ 12846us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.373045 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:13.341400 12725 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:13.244843 14539 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.281424 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:13.358800 14481 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:13.360347 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 77363.5us
I20250113 08:59:13.300644 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:13.284191 14619 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:13.375419 12006 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:13.375972 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 45078.4us
I20250113 08:59:13.384025 12007 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:13.378892 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2165) took 1054 ms. Trace:
I20250113 08:59:13.385578 11968 rpcz_store.cc:276] 0113 08:59:12.324776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.656887 (+332111us) service_pool.cc:225] Handling call
0113 08:59:12.716513 (+ 59626us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.716830 (+ 317us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.716869 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.723511 (+ 6642us) log.cc:844] Serialized 26 byte log entry
0113 08:59:13.276298 (+552787us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:13.276346 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.276436 (+ 90us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:13.378687 (+102251us) raft_consensus.cc:1680] finished
0113 08:59:13.378705 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:13.378837 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":485306,"threads_started":1,"wal-append.queue_time_us":610674}
I20250113 08:59:13.333643 14123 raft_consensus.cc:695] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:13.419994 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:13.422597 11887 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:13.423403 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:13.428220 11885 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:13.431916 12007 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:13.434492 11890 raft_consensus.cc:1235] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20250113 08:59:13.329505 14527 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.433437 12007 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:13.442471 12033 raft_consensus.cc:1235] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:13.346163 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:13.438068 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:13.438668 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:13.444875 12040 raft_consensus.cc:1235] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250113 08:59:13.371336 14590 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:13.374643 14553 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:13.446458 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:13.378098 14497 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:13.449914 12686 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:13.456337 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1653) took 2501 ms. Trace:
I20250113 08:59:13.386085 14391 raft_consensus.cc:695] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:13.469540 12033 rpcz_store.cc:276] 0113 08:59:10.955206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.374345 (+2419139us) service_pool.cc:225] Handling call
0113 08:59:13.442158 (+ 67813us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:13.455895 (+ 13737us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.456193 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:13.450906 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 518) took 2112 ms. Trace:
I20250113 08:59:13.482935 12040 rpcz_store.cc:276] 0113 08:59:11.338039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.381993 (+2043954us) service_pool.cc:225] Handling call
0113 08:59:13.444534 (+ 62541us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:13.450514 (+ 5980us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.450825 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:13.489344 14630 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20250113 08:59:13.491623 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2158) took 2247 ms. Trace:
I20250113 08:59:13.492110 11963 rpcz_store.cc:276] 0113 08:59:11.243901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.920427 (+676526us) service_pool.cc:225] Handling call
0113 08:59:11.920852 (+ 425us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.130920 (+210068us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.130962 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.134615 (+ 3653us) log.cc:844] Serialized 26 byte log entry
0113 08:59:13.232658 (+1098043us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:13.232728 (+ 70us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.232843 (+ 115us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:13.491376 (+258533us) raft_consensus.cc:1680] finished
0113 08:59:13.491398 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:13.491558 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87179,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":7423,"thread_start_us":1267974,"threads_started":2,"wal-append.queue_time_us":1181861}
I20250113 08:59:13.494282 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:13.446219 14342 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:13.476063 14628 log.cc:926] Time spent T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20250113 08:59:13.432054 11959 raft_consensus.cc:2388] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:13.470163 14543 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:13.517146 14543 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.326884 14558 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:13.475025 14627 log.cc:926] Time spent T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250113 08:59:13.474717 14481 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:13.434039 14444 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:13.438571 14604 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:13.335759 14608 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:13.445585 14600 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:13.464111 14553 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:13.478478 14242 consensus_meta.cc:287] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.305s user 0.019s sys 0.003s
I20250113 08:59:13.386632 14629 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.498335 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:13.465556 12036 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:13.361687 14529 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:13.458164 14473 consensus_meta.cc:287] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.907s user 0.007s sys 0.006s
I20250113 08:59:13.467584 14590 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.451100 11889 raft_consensus.cc:1235] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:13.450700 14358 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:13.507670 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:13.509589 14123 consensus_queue.cc:237] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:13.514418 14352 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:13.515110 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 510) took 3256 ms. Trace:
I20250113 08:59:13.431151 14619 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.377892 14438 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.281458 14638 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:13.546015 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2160) took 2084 ms. Trace:
I20250113 08:59:13.528172 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:14.286729 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2164) took 1975 ms. Trace:
I20250113 08:59:14.286579 14638 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.544643 11887 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:14.287178 11961 rpcz_store.cc:276] 0113 08:59:12.311339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.347570 (+ 36231us) service_pool.cc:225] Handling call
0113 08:59:12.371845 (+ 24275us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.464901 (+ 93056us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.464931 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.479247 (+ 14316us) log.cc:844] Serialized 26 byte log entry
0113 08:59:13.233630 (+754383us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:59:13.233673 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.233730 (+ 57us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:14.286505 (+1052775us) raft_consensus.cc:1680] finished
0113 08:59:14.286528 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:14.286655 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11959,"raft.queue_time_us":34353,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":15095,"thread_start_us":701352,"threads_started":2,"wal-append.queue_time_us":753744}
I20250113 08:59:14.288473 14481 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:59:14.289355 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:14.289935 11961 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:13.477403 14581 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:13.615772 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:14.292474 11968 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:14.294209 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 526) took 1286 ms. Trace:
I20250113 08:59:14.294688 11889 rpcz_store.cc:276] 0113 08:59:13.007489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.362565 (+355076us) service_pool.cc:225] Handling call
0113 08:59:13.394905 (+ 32340us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:14.293878 (+898973us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:14.294133 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:13.534390 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:14.299218 12040 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:14.285833 12035 rpcz_store.cc:276] 0113 08:59:10.259034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.570274 (+311240us) service_pool.cc:225] Handling call
0113 08:59:11.769464 (+1199190us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.907742 (+138278us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.514870 (+1607128us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540024ead0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:13.515021 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2704152448}
I20250113 08:59:13.602666 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 697) took 4934 ms. Trace:
I20250113 08:59:14.306907 12722 rpcz_store.cc:276] 0113 08:59:08.667895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.121421 (+2453526us) service_pool.cc:225] Handling call
0113 08:59:11.166218 (+ 44797us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.260686 (+1094468us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.602293 (+1341607us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54006cb954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:13.602583 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2449736448}
I20250113 08:59:14.286156 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 441) took 1113 ms. Trace:
I20250113 08:59:13.575489 12033 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:14.309734 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 519) took 2959 ms. Trace:
I20250113 08:59:14.310132 12033 rpcz_store.cc:276] 0113 08:59:11.350290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.505224 (+2154934us) service_pool.cc:225] Handling call
0113 08:59:13.570989 (+ 65765us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:14.309447 (+738458us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:14.309680 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:13.553455 11960 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:59:14.311978 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 437) took 1140 ms (client timeout 1509 ms). Trace:
I20250113 08:59:14.308415 11965 rpcz_store.cc:276] 0113 08:59:13.173102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.472232 (+299130us) service_pool.cc:225] Handling call
0113 08:59:14.286085 (+813853us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:14.312526 11960 rpcz_store.cc:269] 0113 08:59:13.171919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.222809 (+ 50890us) service_pool.cc:225] Handling call
0113 08:59:14.311903 (+1089094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.314010 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:14.317037 14558 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:13.637768 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 505) took 3103 ms. Trace:
W20250113 08:59:13.874970 14632 log.cc:926] Time spent T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20250113 08:59:14.286911 11966 rpcz_store.cc:276] 0113 08:59:11.461348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.995413 (+534065us) service_pool.cc:225] Handling call
0113 08:59:12.078949 (+ 83536us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.090681 (+ 11732us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.545777 (+1455096us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b5400196c50. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:13.545946 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183899904}
I20250113 08:59:14.320605 11892 rpcz_store.cc:276] 0113 08:59:10.534657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.534952 (+ 295us) service_pool.cc:225] Handling call
0113 08:59:10.535389 (+ 437us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.614146 (+1078757us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:13.615583 (+2001437us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b54001702d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:13.637672 (+ 22089us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2270217472}
I20250113 08:59:14.410101 11960 raft_consensus.cc:2388] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:14.410894 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 435) took 1239 ms. Trace:
I20250113 08:59:14.411412 11960 rpcz_store.cc:276] 0113 08:59:13.171418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.313627 (+1142209us) service_pool.cc:225] Handling call
0113 08:59:14.410803 (+ 97176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.289223 14481 consensus_queue.cc:1046] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250113 08:59:14.222460 14141 consensus_meta.cc:287] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.878s user 0.003s sys 0.011s
I20250113 08:59:14.441288 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:13.703533 14351 consensus_meta.cc:287] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.947s user 0.012s sys 0.002s
W20250113 08:59:13.634899 14458 consensus_meta.cc:287] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.458s user 0.017s sys 0.006s
I20250113 08:59:14.442028 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2181) took 4723 ms. Trace:
I20250113 08:59:13.506850 14342 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:14.453119 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:14.465442 11967 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:14.466204 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 442) took 1264 ms. Trace:
I20250113 08:59:14.466656 11967 rpcz_store.cc:276] 0113 08:59:13.201586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.320570 (+1118984us) service_pool.cc:225] Handling call
0113 08:59:14.466128 (+145558us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.453264 11961 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:14.283031 14543 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:13.642380 14522 raft_consensus.cc:3049] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:14.484760 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:14.486078 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 719) took 1320 ms (client timeout 1515 ms). Trace:
W20250113 08:59:14.486625 12722 rpcz_store.cc:269] 0113 08:59:13.165743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.409697 (+1243954us) service_pool.cc:225] Handling call
0113 08:59:14.485998 (+ 76301us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:13.886911 14625 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:14.457095 12038 rpcz_store.cc:276] 0113 08:59:09.689636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:09.689811 (+ 175us) service_pool.cc:225] Handling call
0113 08:59:09.787584 (+ 97773us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.236091 (+1448507us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:14.412621 (+3176530us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002b9154. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:14.412892 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3220726144}
I20250113 08:59:14.319928 14553 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:14.459431 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:14.504947 11965 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:14.451015 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:14.539377 12033 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:14.283342 14555 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.548318 14619 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:13.640146 14575 raft_consensus.cc:3049] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:13.588485 14562 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:14.549235 14619 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:14.289935 14604 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:14.550350 14604 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:13.589949 14407 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:13.842937 14477 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:13.760004 14552 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.472081 14194 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:13.545652 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:13.499084 14476 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.392542 14352 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:14.476406 11968 raft_consensus.cc:2405] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:14.477069 11961 raft_consensus.cc:1235] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:14.479475 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:14.285497 14242 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:13.562230 14589 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:14.460309 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:14.005776 14633 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:14.456862 14581 raft_consensus.cc:695] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:14.511415 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:14.513935 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 567) took 4404 ms (client timeout 4165 ms). Trace:
W20250113 08:59:14.514331 14637 log.cc:926] Time spent T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.292s user 0.000s sys 0.000s
I20250113 08:59:14.484200 14522 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:14.513262 14625 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.518268 14529 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:13.505723 14601 consensus_queue.cc:237] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.487727 14342 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.454421 14141 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.461519 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:13.814837 14391 consensus_queue.cc:237] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.546269 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 453) took 1118 ms. Trace:
I20250113 08:59:14.628738 12033 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:14.470184 14458 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.547281 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:14.654769 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 718) took 1489 ms (client timeout 1592 ms). Trace:
I20250113 08:59:14.655082 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 450) took 1285 ms. Trace:
W20250113 08:59:14.526664 14344 consensus_meta.cc:287] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.229s user 0.008s sys 0.008s
I20250113 08:59:13.591691 14441 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.553313 12040 raft_consensus.cc:2405] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:14.552284 14575 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:14.567299 14638 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.662854 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1664) took 1233 ms. Trace:
I20250113 08:59:14.663342 12040 rpcz_store.cc:276] 0113 08:59:13.429132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.484021 (+ 54889us) service_pool.cc:225] Handling call
0113 08:59:14.662769 (+1178748us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.601562 11891 raft_consensus.cc:1235] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:14.607739 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 447) took 1328 ms. Trace:
I20250113 08:59:14.608011 12038 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 9.
I20250113 08:59:14.608665 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2169) took 2246 ms. Trace:
I20250113 08:59:14.600829 14271 raft_consensus.cc:3049] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Stepping down as leader of term 2
W20250113 08:59:14.607456 12721 rpcz_store.cc:269] 0113 08:59:10.109602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.640725 (+531123us) service_pool.cc:225] Handling call
0113 08:59:14.511127 (+3870402us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b54006ca7d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:14.513806 (+ 2679us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53063424}
I20250113 08:59:14.665009 12040 raft_consensus.cc:1235] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:14.617435 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 436) took 1445 ms. Trace:
I20250113 08:59:14.667235 11968 rpcz_store.cc:276] 0113 08:59:13.279886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.462386 (+182500us) service_pool.cc:225] Handling call
0113 08:59:14.607645 (+1145259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.668412 11966 rpcz_store.cc:276] 0113 08:59:13.171663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.390052 (+1218389us) service_pool.cc:225] Handling call
0113 08:59:14.617354 (+227302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.652515 11965 rpcz_store.cc:276] 0113 08:59:13.427854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.389215 (+961361us) service_pool.cc:225] Handling call
0113 08:59:14.546176 (+156961us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.456856 14351 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.591550 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2175) took 1054 ms. Trace:
I20250113 08:59:14.549988 14619 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:14.659961 14547 consensus_meta.cc:287] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.482s user 0.012s sys 0.004s
W20250113 08:59:14.655241 12722 rpcz_store.cc:269] 0113 08:59:13.165438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.494257 (+1328819us) service_pool.cc:225] Handling call
0113 08:59:14.654696 (+160439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.659651 12035 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:14.667465 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 536) took 1065 ms. Trace:
I20250113 08:59:14.667413 11961 rpcz_store.cc:276] 0113 08:59:12.362316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.468223 (+2105907us) service_pool.cc:225] Handling call
0113 08:59:14.468576 (+ 353us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:14.608346 (+139770us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:14.608583 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.609508 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 432) took 1438 ms. Trace:
I20250113 08:59:14.709961 12038 rpcz_store.cc:276] 0113 08:59:13.601920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.492578 (+890658us) service_pool.cc:225] Handling call
0113 08:59:14.667404 (+174826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.671095 11965 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20250113 08:59:14.653477 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2217) took 1034 ms. Trace:
I20250113 08:59:14.711068 12033 rpcz_store.cc:276] 0113 08:59:13.618835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.311231 (+692396us) service_pool.cc:225] Handling call
0113 08:59:14.653401 (+342170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.674088 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 520) took 3296 ms. Trace:
I20250113 08:59:14.711947 12040 rpcz_store.cc:276] 0113 08:59:11.378156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.664305 (+3286149us) service_pool.cc:225] Handling call
0113 08:59:14.664731 (+ 426us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:14.673734 (+ 9003us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:14.674006 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.585721 14629 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:14.655885 11960 rpcz_store.cc:276] 0113 08:59:13.369866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.412187 (+1042321us) service_pool.cc:225] Handling call
0113 08:59:14.654994 (+242807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.670184 14271 raft_consensus.cc:738] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:14.592789 14608 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:14.676420 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.296s user 0.013s sys 0.028s
I20250113 08:59:14.710093 11967 rpcz_store.cc:276] 0113 08:59:13.170603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.467579 (+1296976us) service_pool.cc:225] Handling call
0113 08:59:14.609342 (+141763us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.683388 11962 raft_consensus.cc:2388] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:14.716418 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 434) took 1545 ms. Trace:
W20250113 08:59:14.719238 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.339s user 0.013s sys 0.028s
I20250113 08:59:14.719383 11965 rpcz_store.cc:276] 0113 08:59:13.171147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.670036 (+1498889us) service_pool.cc:225] Handling call
0113 08:59:14.670696 (+ 660us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:14.716047 (+ 45351us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:14.716339 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:14.719815 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.33955e+06us
I20250113 08:59:14.594758 14352 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.717765 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:14.623185 14589 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.723631 14271 consensus_queue.cc:260] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.725456 14271 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:14.727201 14352 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:14.724159 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 717) took 1559 ms (client timeout 1724 ms). Trace:
I20250113 08:59:14.670573 14344 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:14.729408 11961 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 6. Current term is 7. Ops: [6.8-6.8]
W20250113 08:59:14.568323 14539 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.022s user 0.003s sys 0.011s
I20250113 08:59:14.742085 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 445) took 1497 ms. Trace:
I20250113 08:59:14.742496 11961 rpcz_store.cc:276] 0113 08:59:13.244909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.711201 (+1466292us) service_pool.cc:225] Handling call
0113 08:59:14.728908 (+ 17707us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:14.741730 (+ 12822us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:14.742007 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.708328 14476 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.744448 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:14.746845 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 588) took 1512 ms (client timeout 1754 ms). Trace:
W20250113 08:59:14.747203 12685 connection.cc:232] RPC call timeout handler was delayed by -1.1509269919999952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2772s for a response
W20250113 08:59:14.747226 12722 rpcz_store.cc:269] 0113 08:59:13.234861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.717297 (+1482436us) service_pool.cc:225] Handling call
0113 08:59:14.746788 (+ 29491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.722011 14629 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:14.745002 14591 consensus_meta.cc:287] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.565s user 0.011s sys 0.006s
I20250113 08:59:14.677151 14522 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.764834 14522 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:14.764950 14351 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.765882 14129 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:14.680609 11959 rpcz_store.cc:276] 0113 08:59:13.536937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.545198 (+ 8261us) service_pool.cc:225] Handling call
0113 08:59:14.591456 (+1046258us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:14.745692 14539 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.200s user 0.003s sys 0.012s
I20250113 08:59:14.716528 14547 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.649925 14633 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.750072 14352 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:14.738433 12721 rpcz_store.cc:269] 0113 08:59:13.165068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.675600 (+1510532us) service_pool.cc:225] Handling call
0113 08:59:14.724095 (+ 48495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:14.766568 14342 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:14.772832 14618 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.311s user 0.011s sys 0.032s
I20250113 08:59:14.784888 14589 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.744861 14344 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.728453 14581 consensus_queue.cc:237] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.744180 14575 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.856806 14458 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:14.873158 12008 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.426s user 0.016s sys 0.042s
W20250113 08:59:14.873579 12008 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.426s user 0.016s sys 0.042s
W20250113 08:59:14.848408 14618 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.387s user 0.012s sys 0.032s
I20250113 08:59:14.841643 14553 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:14.767264 14591 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:14.883548 12007 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.441s user 0.015s sys 0.043s
I20250113 08:59:14.748425 12685 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:14.857308 14625 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:14.767035 14629 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:14.887418 12007 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.445s user 0.015s sys 0.043s
W20250113 08:59:14.896654 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.46307e+06us
W20250113 08:59:14.830924 14549 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.198s user 0.009s sys 0.005s
W20250113 08:59:14.887902 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 139591us
W20250113 08:59:14.874495 12718 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.497s user 0.094s sys 0.184s
I20250113 08:59:14.873467 14242 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:14.833163 14352 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:14.886168 14455 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.529s user 0.038s sys 0.081s
W20250113 08:59:14.901897 14587 consensus_meta.cc:287] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.612s user 0.014s sys 0.003s
I20250113 08:59:15.060766 14633 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:14.956928 14635 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.481s user 0.011s sys 0.016s
I20250113 08:59:15.041033 14619 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.044204 14629 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:15.144171 14635 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.667s user 0.013s sys 0.018s
W20250113 08:59:15.198222 14624 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.576s user 0.013s sys 0.043s
W20250113 08:59:15.200102 14624 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.630s user 0.013s sys 0.043s
W20250113 08:59:15.196131 14610 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.675s user 0.017s sys 0.043s
W20250113 08:59:15.220580 14610 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.765s user 0.018s sys 0.044s
W20250113 08:59:15.225174 14623 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.001s sys 0.047s
I20250113 08:59:15.242167 14644 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election won for term 6
W20250113 08:59:15.249776 14623 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.000s user 0.002s sys 0.047s
I20250113 08:59:15.283325 14644 raft_consensus.cc:695] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:15.299167 14610 raft_consensus.cc:3049] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:15.300551 14610 raft_consensus.cc:738] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:15.303248 14610 consensus_queue.cc:260] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.305657 14644 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.307687 14392 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:15.304649 14610 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:15.330220 12042 raft_consensus.cc:2405] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250113 08:59:15.482528 11932 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.113s user 0.016s sys 0.015s
W20250113 08:59:15.483652 11932 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.114s user 0.017s sys 0.016s
W20250113 08:59:15.485474 11932 connection.cc:232] RPC call timeout handler was delayed by -2.0241508249998788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
W20250113 08:59:15.485997 11932 connection.cc:232] RPC call timeout handler was delayed by -2.0199364729999161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1786s for a response
W20250113 08:59:15.486366 11932 connection.cc:232] RPC call timeout handler was delayed by -1.8314281680000022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20250113 08:59:15.486696 11932 connection.cc:232] RPC call timeout handler was delayed by -1.8221965609999415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
W20250113 08:59:15.487721 11932 connection.cc:232] RPC call timeout handler was delayed by -1.7542829399999391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20099999999999998s for a response
W20250113 08:59:15.487949 11932 connection.cc:232] RPC call timeout handler was delayed by -1.7451508249998824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21400000000000002s for a response
W20250113 08:59:15.488160 11932 connection.cc:232] RPC call timeout handler was delayed by -1.4936965609999788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20250113 08:59:15.488375 11932 connection.cc:232] RPC call timeout handler was delayed by -1.3460965609999676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21050000000000002s for a response
I20250113 08:59:15.493569 11932 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:15.494911 14664 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:15.497715 11932 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:15.509109 14665 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:15.510365 14665 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:14.903796 14141 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:15.512105 14271 consensus_meta.cc:287] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.785s user 0.015s sys 0.006s
I20250113 08:59:15.513710 14665 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:14.981616 11959 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:15.515161 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 452) took 2087 ms. Trace:
I20250113 08:59:15.515477 11959 rpcz_store.cc:276] 0113 08:59:13.427488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.816077 (+1388589us) service_pool.cc:225] Handling call
0113 08:59:14.887219 (+ 71142us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.514557 (+627338us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.514929 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.519083 14664 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:15.523610 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2174) took 2003 ms. Trace:
I20250113 08:59:15.524773 11959 rpcz_store.cc:276] 0113 08:59:13.520604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.516516 (+1995912us) service_pool.cc:225] Handling call
0113 08:59:15.517323 (+ 807us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.518061 (+ 738us) raft_consensus.cc:1529] Early marking committed up to index 3
0113 08:59:15.520375 (+ 2314us) log.cc:844] Serialized 14 byte log entry
0113 08:59:15.523137 (+ 2762us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:15.523185 (+ 48us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:15.523258 (+ 73us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.523349 (+ 91us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.523506 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":230,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":79,"raft.queue_time_us":168,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"spinlock_wait_cycles":70528,"thread_start_us":3925,"threads_started":2,"wal-append.queue_time_us":2668}
I20250113 08:59:15.532771 14467 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.533468 14467 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:14.926936 14549 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.294s user 0.009s sys 0.005s
I20250113 08:59:15.029891 14553 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:15.580526 11931 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.131s user 0.017s sys 0.011s
W20250113 08:59:15.581012 11931 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.131s user 0.017s sys 0.011s
W20250113 08:59:15.581504 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 2.13207e+06us
I20250113 08:59:15.584211 11931 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:15.585487 14665 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:15.586714 11931 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:14.975713 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 77758.6us
I20250113 08:59:15.587999 14665 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:15.588831 14665 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:15.589430 14665 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:15.598471 14391 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.012s sys 0.015s
I20250113 08:59:15.603857 14406 raft_consensus.cc:3049] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:15.605803 14647 consensus_queue.cc:1035] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
W20250113 08:59:15.606716 14391 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.012s sys 0.017s
I20250113 08:59:15.608111 14646 consensus_queue.cc:1046] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250113 08:59:15.612169 14438 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.606897 14406 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:15.612984 14438 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.614832 14406 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.615680 14587 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.616590 14406 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:15.617319 14438 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.620282 14612 raft_consensus.cc:3049] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:15.620841 14612 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:15.621150 14587 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.622283 14646 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.623170 14646 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.623867 14612 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.624485 14665 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.625661 14467 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:15.625813 14612 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:15.627702 14665 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.625694 14646 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:15.633855 14441 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:15.634536 12007 connection.cc:232] RPC call timeout handler was delayed by -2.2570292220000283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20250113 08:59:15.634779 14441 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.636103 14612 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.636255 14646 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.636981 14612 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.637100 14646 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:15.638301 14129 consensus_meta.cc:287] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.872s user 0.008s sys 0.011s
I20250113 08:59:15.639284 14612 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:15.639333 14646 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:15.117993 14449 raft_consensus.cc:3049] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:15.640770 14449 raft_consensus.cc:738] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:15.642079 14441 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:15.642000 14449 consensus_queue.cc:260] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.643497 14449 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:15.658885 14665 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.659790 14665 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.662210 11959 raft_consensus.cc:1235] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:15.662297 14665 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.663682 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:15.664114 14449 consensus_queue.cc:1046] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:15.664566 11959 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:15.666268 12007 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:15.666488 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:15.667800 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:15.668429 11959 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:15.668817 14449 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:15.669088 12687 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.857s user 0.018s sys 0.010s
I20250113 08:59:15.669358 14449 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:15.669834 14449 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:15.669584 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:15.671303 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:15.671957 11959 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:15.672835 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 584) took 3351 ms (client timeout 2667 ms). Trace:
I20250113 08:59:15.673338 11959 raft_consensus.cc:1235] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:15.673297 12725 rpcz_store.cc:269] 0113 08:59:12.321491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.352766 (+ 31275us) service_pool.cc:225] Handling call
0113 08:59:15.672305 (+3319539us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b54007a17d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.672563 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2250348416}
I20250113 08:59:15.674719 14441 consensus_queue.cc:1046] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:15.675256 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:15.675918 14467 raft_consensus.cc:3049] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:15.676262 12725 raft_consensus.cc:2388] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:15.676402 14467 raft_consensus.cc:738] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:15.676802 12725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 722) took 2428 ms (client timeout 2927 ms). Trace:
W20250113 08:59:15.669557 12687 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.909s user 0.019s sys 0.010s
W20250113 08:59:15.678423 12718 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.308s user 0.104s sys 0.192s
I20250113 08:59:15.678251 14467 consensus_queue.cc:260] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.679831 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 572) took 5106 ms. Trace:
I20250113 08:59:15.680303 12718 rpcz_store.cc:276] 0113 08:59:10.572992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.598092 (+2025100us) service_pool.cc:225] Handling call
0113 08:59:12.705778 (+107686us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:12.742462 (+ 36684us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.743439 (+ 977us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:12.756972 (+ 13533us) log.cc:844] Serialized 26 byte log entry
0113 08:59:13.038196 (+281224us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:13.333506 (+295310us) log.cc:844] Serialized 14 byte log entry
0113 08:59:15.679258 (+2345752us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.679416 (+ 158us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:15.679504 (+ 88us) raft_consensus.cc:1680] finished
0113 08:59:15.679528 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.679744 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10614,"prepare.queue_time_us":1082268,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":62969,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":254,"thread_start_us":2685258,"threads_started":3,"wal-append.queue_time_us":493750}
W20250113 08:59:15.677186 12725 rpcz_store.cc:269] 0113 08:59:13.247965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.674616 (+2426651us) service_pool.cc:225] Handling call
0113 08:59:15.676726 (+ 2110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.683473 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:15.684612 12718 raft_consensus.cc:2388] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:59:15.687053 11965 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.003s sys 0.011s
W20250113 08:59:15.687409 11965 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.003s sys 0.012s
I20250113 08:59:15.684538 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:15.692247 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2162) took 3432 ms (client timeout 1892 ms). Trace:
W20250113 08:59:15.692657 11962 rpcz_store.cc:269] 0113 08:59:12.259956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.272380 (+ 12424us) service_pool.cc:225] Handling call
0113 08:59:15.691711 (+3419331us) spinlock_profiling.cc:243] Waited 2.4 s on lock 0x7b540006dbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.692027 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5515854080}
I20250113 08:59:15.692414 12725 raft_consensus.cc:2405] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:15.689033 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 449) took 2346 ms. Trace:
I20250113 08:59:15.696283 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 578) took 4473 ms. Trace:
W20250113 08:59:15.697065 14407 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.014s user 0.005s sys 0.013s
W20250113 08:59:15.697628 14407 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.019s user 0.006s sys 0.013s
I20250113 08:59:15.696756 12725 rpcz_store.cc:276] 0113 08:59:11.223293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.695016 (+4471723us) service_pool.cc:225] Handling call
0113 08:59:15.695516 (+ 500us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.695960 (+ 444us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:15.696012 (+ 52us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:15.696073 (+ 61us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:15.696116 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.696159 (+ 43us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.696226 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.696066 11965 rpcz_store.cc:276] 0113 08:59:13.342363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.720258 (+1377895us) service_pool.cc:225] Handling call
0113 08:59:14.744580 (+ 24322us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:14.744948 (+ 368us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:14.744978 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:14.749920 (+ 4942us) log.cc:844] Serialized 26 byte log entry
0113 08:59:15.687920 (+938000us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:15.688395 (+ 475us) log.cc:844] Serialized 14 byte log entry
0113 08:59:15.688647 (+ 252us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.688717 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:15.688781 (+ 64us) raft_consensus.cc:1680] finished
0113 08:59:15.688794 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.688958 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1178,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":957,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":3641,"thread_start_us":647144,"threads_started":1,"wal-append.queue_time_us":552916}
I20250113 08:59:15.685384 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 587) took 2518 ms. Trace:
I20250113 08:59:15.699095 12718 rpcz_store.cc:276] 0113 08:59:13.167148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.682967 (+2515819us) service_pool.cc:225] Handling call
0113 08:59:15.685324 (+ 2357us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.700101 12725 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250113 08:59:15.701315 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 797) took 4361 ms. Trace:
I20250113 08:59:15.701670 12725 rpcz_store.cc:276] 0113 08:59:11.340080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.698917 (+4358837us) service_pool.cc:225] Handling call
0113 08:59:15.699685 (+ 768us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:15.700991 (+ 1306us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.701206 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.703239 14449 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.703851 12725 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:15.705286 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 581) took 3574 ms. Trace:
I20250113 08:59:15.705729 12725 rpcz_store.cc:276] 0113 08:59:12.130720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.703081 (+3572361us) service_pool.cc:225] Handling call
0113 08:59:15.703457 (+ 376us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.705021 (+ 1564us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.705202 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.706315 14449 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:15.701556 12718 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250113 08:59:15.707065 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:15.707687 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 802) took 4279 ms. Trace:
I20250113 08:59:15.707832 11959 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:15.706358 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 571) took 5138 ms (client timeout 2127 ms). Trace:
W20250113 08:59:15.708734 12719 rpcz_store.cc:269] 0113 08:59:10.567564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.568232 (+ 668us) service_pool.cc:225] Handling call
0113 08:59:15.705921 (+5137689us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54006bdad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.706173 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4234071936}
W20250113 08:59:15.710254 14455 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 2.353s user 0.039s sys 0.095s
I20250113 08:59:15.708088 12718 rpcz_store.cc:276] 0113 08:59:11.428127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.700642 (+4272515us) service_pool.cc:225] Handling call
0113 08:59:15.701131 (+ 489us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:15.707350 (+ 6219us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.707585 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.711643 12725 raft_consensus.cc:1235] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250113 08:59:15.712457 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 716) took 3545 ms. Trace:
I20250113 08:59:15.712838 12725 rpcz_store.cc:276] 0113 08:59:12.166996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.707043 (+3540047us) service_pool.cc:225] Handling call
0113 08:59:15.711191 (+ 4148us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:15.712171 (+ 980us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.712394 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.713845 14591 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.715955 12725 raft_consensus.cc:1235] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:15.040385 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:15.718072 12722 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:15.718714 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 723) took 2257 ms (client timeout 1633 ms). Trace:
W20250113 08:59:15.719156 12722 rpcz_store.cc:269] 0113 08:59:13.461471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.751604 (+1290133us) service_pool.cc:225] Handling call
0113 08:59:15.718574 (+966970us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.720726 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 585) took 3365 ms. Trace:
I20250113 08:59:15.721189 12718 rpcz_store.cc:276] 0113 08:59:12.355377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.711810 (+3356433us) service_pool.cc:225] Handling call
0113 08:59:15.712939 (+ 1129us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:15.714919 (+ 1980us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:15.714966 (+ 47us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:15.715747 (+ 781us) log.cc:844] Serialized 26 byte log entry
0113 08:59:15.718209 (+ 2462us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:15.718248 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.718336 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:15.720476 (+ 2140us) raft_consensus.cc:1680] finished
0113 08:59:15.720502 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.720648 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":206,"thread_start_us":1668,"threads_started":1,"wal-append.queue_time_us":2202}
W20250113 08:59:15.723176 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 803) took 4294 ms (client timeout 1654 ms). Trace:
W20250113 08:59:15.723783 12720 rpcz_store.cc:269] 0113 08:59:11.428395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.444755 (+ 16360us) service_pool.cc:225] Handling call
0113 08:59:15.722593 (+4277838us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54006004d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.722893 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":328154368}
I20250113 08:59:15.720988 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 721) took 2554 ms. Trace:
I20250113 08:59:15.716708 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 720) took 2550 ms. Trace:
I20250113 08:59:15.728636 12722 rpcz_store.cc:276] 0113 08:59:13.166570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.719874 (+2553304us) service_pool.cc:225] Handling call
0113 08:59:15.720299 (+ 425us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.720650 (+ 351us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:15.720674 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:15.720737 (+ 63us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:15.720774 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.720828 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.720940 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.730046 12726 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250113 08:59:15.729043 12725 rpcz_store.cc:276] 0113 08:59:13.166165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.714155 (+2547990us) service_pool.cc:225] Handling call
0113 08:59:15.715574 (+ 1419us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.716432 (+ 858us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.716632 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.730530 12726 pending_rounds.cc:77] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting all ops after (but not including) 5
I20250113 08:59:15.730871 12726 pending_rounds.cc:99] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250113 08:59:15.731209 12726 raft_consensus.cc:2883] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:15.728763 11959 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:15.732565 12007 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:15.733410 14612 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:15.733973 14612 raft_consensus.cc:695] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:15.735039 14612 consensus_queue.cc:237] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:15.737674 14558 consensus_meta.cc:287] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.420s user 0.011s sys 0.008s
I20250113 08:59:15.739423 14591 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:15.740432 14591 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.743492 14591 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.085275 14547 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:15.756107 14575 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:15.030177 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1915) took 3807 ms. Trace:
W20250113 08:59:15.761111 14604 consensus_meta.cc:287] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 1.210s user 0.019s sys 0.000s
I20250113 08:59:15.761768 14604 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.760874 11893 rpcz_store.cc:276] 0113 08:59:11.223168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.025671 (+802503us) service_pool.cc:225] Handling call
0113 08:59:12.031393 (+ 5722us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.150778 (+119385us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.029844 (+2879066us) spinlock_profiling.cc:243] Waited 94 ms on lock 0x7b540038e854. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:15.030095 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216184576}
I20250113 08:59:15.764061 14604 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:15.766520 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:15.767674 11959 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 4: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:15.771924 12038 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:15.781699 14344 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:15.798130 11886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 513) took 4714 ms (client timeout 2772 ms). Trace:
W20250113 08:59:15.798689 11886 rpcz_store.cc:269] 0113 08:59:11.083274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.083418 (+ 144us) service_pool.cc:225] Handling call
0113 08:59:15.797469 (+4714051us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b540026d254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.797757 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":827907328}
W20250113 08:59:15.802956 12034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1635) took 5210 ms (client timeout 2029 ms). Trace:
I20250113 08:59:15.803254 12037 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250113 08:59:15.803700 12037 pending_rounds.cc:77] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 5
W20250113 08:59:15.803484 12034 rpcz_store.cc:269] 0113 08:59:10.592663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.113356 (+520693us) service_pool.cc:225] Handling call
0113 08:59:15.802317 (+4688961us) spinlock_profiling.cc:243] Waited 68.1 ms on lock 0x7b5400248954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.802614 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":156540288}
I20250113 08:59:15.804150 12037 pending_rounds.cc:99] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250113 08:59:15.804481 12037 raft_consensus.cc:2883] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:15.806041 12034 raft_consensus.cc:1235] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:15.806682 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2198) took 3464 ms. Trace:
I20250113 08:59:15.806490 14407 consensus_queue.cc:1046] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:15.806804 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1650) took 5066 ms. Trace:
I20250113 08:59:15.807442 14407 consensus_queue.cc:1225] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20250113 08:59:15.894848 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.662s user 0.008s sys 0.027s
W20250113 08:59:15.906292 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.673s user 0.008s sys 0.035s
I20250113 08:59:15.881374 11859 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:15.843501 11965 raft_consensus.cc:1270] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250113 08:59:15.908254 14547 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:15.908762 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:15.909088 14613 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:15.909922 12725 raft_consensus.cc:2405] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:15.911861 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 4.3771e+06us
W20250113 08:59:15.913175 12035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 535) took 2391 ms (client timeout 1538 ms). Trace:
I20250113 08:59:15.901122 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2158) took 4549 ms. Trace:
W20250113 08:59:15.913686 12035 rpcz_store.cc:269] 0113 08:59:13.521331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.306520 (+785189us) service_pool.cc:225] Handling call
0113 08:59:15.912709 (+1606189us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b540020d5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.912940 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":482345216}
I20250113 08:59:15.915278 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:15.915643 14560 raft_consensus.cc:3049] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:15.914551 11964 rpcz_store.cc:276] 0113 08:59:11.351265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.979751 (+628486us) service_pool.cc:225] Handling call
0113 08:59:11.992754 (+ 13003us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:12.309191 (+316437us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:12.309220 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:12.325808 (+ 16588us) log.cc:844] Serialized 52 byte log entry
0113 08:59:13.180935 (+855127us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:13.180971 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.900125 (+2719154us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54002f0654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:15.900243 (+ 118us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:15.900356 (+ 113us) raft_consensus.cc:1680] finished
0113 08:59:15.900374 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.900876 (+ 502us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36354,"raft.queue_time_us":48421,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":41199,"spinlock_wait_cycles":470766848,"thread_start_us":786264,"threads_started":1,"wal-append.queue_time_us":889766,"wal-append.run_cpu_time_us":13757,"wal-append.run_wall_time_us":2168674}
I20250113 08:59:15.821488 11967 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:15.913714 14547 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250113 08:59:15.905524 12033 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.017s sys 0.011s
W20250113 08:59:15.917578 12033 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.166s user 0.017s sys 0.011s
I20250113 08:59:15.916115 14560 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:15.918643 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 525) took 3839 ms. Trace:
I20250113 08:59:15.919003 12719 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:15.919528 14560 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.919090 12033 rpcz_store.cc:276] 0113 08:59:12.079320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.719602 (+2640282us) service_pool.cc:225] Handling call
0113 08:59:14.719994 (+ 392us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:14.720321 (+ 327us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:14.720358 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:14.729948 (+ 9590us) log.cc:844] Serialized 26 byte log entry
0113 08:59:15.918190 (+1188242us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:15.918309 (+ 119us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.918394 (+ 85us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:15.918432 (+ 38us) raft_consensus.cc:1680] finished
0113 08:59:15.918441 (+ 9us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.918563 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1165957,"threads_started":1,"wal-append.queue_time_us":555648}
I20250113 08:59:14.975462 11963 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:15.921547 14352 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:15.921627 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:15.841370 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 702) took 7031 ms. Trace:
I20250113 08:59:15.922858 12724 rpcz_store.cc:276] 0113 08:59:08.809923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.242355 (+2432432us) service_pool.cc:225] Handling call
0113 08:59:11.257935 (+ 15580us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:11.644892 (+386957us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.840878 (+4195986us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b54007e0ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:15.841127 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":888492288}
W20250113 08:59:15.839838 14497 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.197s user 0.003s sys 0.024s
I20250113 08:59:15.884858 14527 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:15.924537 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2172) took 2670 ms. Trace:
I20250113 08:59:12.571822 12723 rpcz_store.cc:276] 0113 08:59:08.843672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.083084 (+3239412us) service_pool.cc:225] Handling call
0113 08:59:12.083400 (+ 316us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.083648 (+ 248us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:12.361561 (+277913us) log.cc:844] Serialized 14 byte log entry
0113 08:59:12.494660 (+133099us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:12.494718 (+ 58us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:12.494759 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:12.494841 (+ 82us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:12.494960 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14439,"prepare.queue_time_us":394641,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":50086,"spinlock_wait_cycles":98344320,"thread_start_us":386562,"threads_started":2}
W20250113 08:59:15.924716 14497 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.289s user 0.004s sys 0.029s
I20250113 08:59:14.934082 11966 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:15.926750 14467 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:15.807319 12037 rpcz_store.cc:276] 0113 08:59:10.740181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.617390 (+1877209us) service_pool.cc:225] Handling call
0113 08:59:15.802594 (+3185204us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.805083 (+ 2489us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.806546 (+ 1463us) spinlock_profiling.cc:243] Waited 3.18 s on lock 0x7b5400248950. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:15.806728 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7324512000}
I20250113 08:59:15.925400 11960 rpcz_store.cc:276] 0113 08:59:13.254292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.715736 (+1461444us) service_pool.cc:225] Handling call
0113 08:59:14.740064 (+ 24328us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:14.848826 (+108762us) raft_consensus.cc:1529] Early marking committed up to index 2
0113 08:59:15.519220 (+670394us) log.cc:844] Serialized 14 byte log entry
0113 08:59:15.521447 (+ 2227us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:15.521530 (+ 83us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:15.521683 (+ 153us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.924180 (+402497us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54002a3354. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:15.924222 (+ 42us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:15.924338 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91,"prepare.queue_time_us":669027,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":121,"raft.queue_time_us":746,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":241,"spinlock_wait_cycles":249284480,"thread_start_us":475288,"threads_started":2,"wal-append.queue_time_us":2044}
I20250113 08:59:15.923797 11859 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:15.928220 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2171) took 3174 ms. Trace:
I20250113 08:59:15.928229 14527 raft_consensus.cc:2743] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:15.922346 14636 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.928411 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:15.930835 14636 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:15.921860 12686 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.268s user 0.010s sys 0.012s
W20250113 08:59:15.932296 12686 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.279s user 0.011s sys 0.013s
I20250113 08:59:15.922801 12034 rpcz_store.cc:276] 0113 08:59:12.342422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.804946 (+3462524us) service_pool.cc:225] Handling call
0113 08:59:15.805600 (+ 654us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.806436 (+ 836us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.806604 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.933948 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:15.934619 12686 connection.cc:232] RPC call timeout handler was delayed by -2.1942768589999559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
I20250113 08:59:15.932194 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:15.935945 12686 connection.cc:232] RPC call timeout handler was delayed by -1.8549768589999758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1886s for a response
I20250113 08:59:15.936553 12037 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:14.975282 14573 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.436s user 0.004s sys 0.028s
I20250113 08:59:15.939033 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:15.939472 14573 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 2.420s user 0.008s sys 0.032s
I20250113 08:59:15.940606 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:15.916129 12035 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:15.932519 14341 raft_consensus.cc:3049] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:15.943663 14341 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:15.943711 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:15.936470 12686 connection.cc:232] RPC call timeout handler was delayed by -1.8414768589999539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
W20250113 08:59:15.945042 12686 connection.cc:232] RPC call timeout handler was delayed by -1.8162768590000269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20250113 08:59:15.945362 12686 connection.cc:232] RPC call timeout handler was delayed by -1.7739768589999585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
I20250113 08:59:15.941035 14497 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:15.947249 12684 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.179s user 0.014s sys 0.015s
I20250113 08:59:15.947786 11859 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:15.948837 14527 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:15.947110 14497 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.037611 14561 raft_consensus.cc:3049] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:15.951319 14497 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:15.951746 14561 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:15.945765 12686 connection.cc:232] RPC call timeout handler was delayed by -1.7199768589999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
I20250113 08:59:15.945061 14341 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.953814 12035 raft_consensus.cc:2388] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:15.954787 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:15.955325 12686 connection.cc:232] RPC call timeout handler was delayed by -1.6956768589999456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20250113 08:59:15.947831 12684 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.180s user 0.015s sys 0.016s
W20250113 08:59:15.956087 12686 connection.cc:232] RPC call timeout handler was delayed by -1.6659768590000112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2096s for a response
W20250113 08:59:15.956660 12686 connection.cc:232] RPC call timeout handler was delayed by -1.6515107190000435s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
W20250113 08:59:15.957139 12686 connection.cc:232] RPC call timeout handler was delayed by -1.18537685900003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.263s for a response
I20250113 08:59:15.935313 12034 raft_consensus.cc:2405] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:15.957643 12686 connection.cc:232] RPC call timeout handler was delayed by -1.057576858999937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2772s for a response
I20250113 08:59:15.958977 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:15.955085 14561 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.959879 12686 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:15.960572 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:15.961386 14561 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:15.962618 12686 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:15.965160 14562 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:15.967392 14539 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250113 08:59:15.968102 12684 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 1.936s (SENT) [suppressed 10 similar messages]
I20250113 08:59:15.972128 12035 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:15.922446 12033 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:15.972540 14481 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.435s user 0.022s sys 0.025s
I20250113 08:59:15.928682 11966 rpcz_store.cc:276] 0113 08:59:12.753843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.669492 (+1915649us) service_pool.cc:225] Handling call
0113 08:59:14.718406 (+ 48914us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.926237 (+1207831us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:15.927827 (+ 1590us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54001ae5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:15.928049 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":495523712}
W20250113 08:59:15.976331 14481 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.439s user 0.022s sys 0.025s
I20250113 08:59:15.977418 14537 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:15.973234 14562 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:15.808769 12685 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:15.982244 12037 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:15.984931 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:14.933526 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:15.987974 12686 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:15.988981 14539 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:15.990114 12686 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:15.989034 14537 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:15.991624 14537 raft_consensus.cc:695] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:15.993526 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 586) took 2826 ms (client timeout 1972 ms). Trace:
W20250113 08:59:15.993964 12721 rpcz_store.cc:269] 0113 08:59:13.166897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.869891 (+1702994us) service_pool.cc:225] Handling call
0113 08:59:15.988371 (+1118480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:15.994594 14539 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:15.994849 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:15.995364 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.058s user 0.011s sys 0.012s
W20250113 08:59:15.989004 14601 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.222s user 0.010s sys 0.009s
W20250113 08:59:15.996126 14601 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.230s user 0.010s sys 0.009s
I20250113 08:59:15.997431 14539 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:15.998106 14539 raft_consensus.cc:695] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:15.999388 14539 consensus_queue.cc:237] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:15.039950 11961 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:16.001308 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 451) took 2574 ms. Trace:
I20250113 08:59:16.001628 11961 rpcz_store.cc:276] 0113 08:59:13.427042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.743615 (+1316573us) service_pool.cc:225] Handling call
0113 08:59:15.039712 (+296097us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.000787 (+961075us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.001112 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.002215 11962 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:16.006974 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 515) took 5717 ms. Trace:
I20250113 08:59:16.007431 12039 rpcz_store.cc:276] 0113 08:59:10.289743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.333342 (+2043599us) service_pool.cc:225] Handling call
0113 08:59:12.333689 (+ 347us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:13.369067 (+1035378us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.006430 (+2637363us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003bcc54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.006779 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2360201088}
W20250113 08:59:16.007951 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.071s user 0.011s sys 0.013s
W20250113 08:59:14.976301 12036 consensus_meta.cc:287] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.557s user 0.005s sys 0.006s
I20250113 08:59:16.008829 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.011405 12036 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:16.012872 14551 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:15.988682 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 180554us
I20250113 08:59:16.019299 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.022382 14527 consensus_queue.cc:1046] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:16.023177 12721 raft_consensus.cc:2388] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:16.023301 12040 raft_consensus.cc:1235] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:16.023522 14527 raft_consensus.cc:3049] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:59:16.022413 14551 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.016022 14601 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.019294 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 583) took 3790 ms. Trace:
I20250113 08:59:16.028450 12719 rpcz_store.cc:276] 0113 08:59:12.226846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:15.710366 (+3483520us) service_pool.cc:225] Handling call
0113 08:59:15.710885 (+ 519us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.919588 (+208703us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.013068 (+ 93480us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b5400741654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.017399 (+ 4331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":477563776}
W20250113 08:59:16.031337 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 63281.4us
I20250113 08:59:16.031481 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.031057 11964 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:16.024034 14527 raft_consensus.cc:738] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:16.024055 14585 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
W20250113 08:59:16.036212 12036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2216) took 2819 ms (client timeout 2109 ms). Trace:
W20250113 08:59:16.039090 12036 rpcz_store.cc:269] 0113 08:59:13.208690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.283796 (+ 75106us) service_pool.cc:225] Handling call
0113 08:59:16.026255 (+2742459us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.040460 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.039863 14527 consensus_queue.cc:260] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:16.042456 12684 connection.cc:232] RPC call timeout handler was delayed by -1.0578309979999858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15380000000000002s for a response
I20250113 08:59:16.043009 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.046489 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.046928 12723 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:16.047390 12724 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:16.048301 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:16.048606 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 522) took 4133 ms (client timeout 1936 ms). Trace:
I20250113 08:59:15.993046 14537 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.035009 14601 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.035225 11856 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.043012 14352 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.055153 14609 consensus_queue.cc:1035] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250113 08:59:16.063895 14590 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.052767 12034 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
W20250113 08:59:16.049016 12041 rpcz_store.cc:269] 0113 08:59:11.915055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.167975 (+252920us) service_pool.cc:225] Handling call
0113 08:59:16.048285 (+3880310us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540002d5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:16.048448 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2340171648}
I20250113 08:59:16.067205 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.067701 11959 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:16.069025 12037 raft_consensus.cc:2405] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:16.052925 14527 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:16.069795 12007 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:16.070962 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.071063 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.053735 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 570) took 5684 ms. Trace:
I20250113 08:59:16.073417 14590 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:16.074252 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.058780 14551 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.074470 14590 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.073621 14488 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:16.075168 12724 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:16.076647 14590 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.073415 12717 rpcz_store.cc:276] 0113 08:59:10.369480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.284080 (+1914600us) service_pool.cc:225] Handling call
0113 08:59:12.284402 (+ 322us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:12.721767 (+437365us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.053237 (+3331470us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54004105d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.053499 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326210816}
I20250113 08:59:16.064498 14352 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:16.085646 11857 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.756s user 0.003s sys 0.019s
I20250113 08:59:16.088814 12719 raft_consensus.cc:2388] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:16.089036 14123 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.523s user 0.015s sys 0.019s
W20250113 08:59:16.089500 14123 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.523s user 0.016s sys 0.019s
I20250113 08:59:16.089669 14488 raft_consensus.cc:695] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:16.088815 11857 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.759s user 0.003s sys 0.020s
I20250113 08:59:16.090338 14522 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.091181 11968 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250113 08:59:16.091473 14551 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:16.089696 14297 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.091086 14522 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.092468 14297 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.093206 14558 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.093758 14558 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.094280 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2170) took 3341 ms. Trace:
I20250113 08:59:16.091245 14547 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.094870 14297 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.095167 14547 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.095815 14558 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:16.096416 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.096771 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.097163 11886 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:16.096074 14551 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.097975 14547 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:16.094215 14522 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.098763 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 524) took 3092 ms. Trace:
I20250113 08:59:16.099323 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.091037 14488 consensus_queue.cc:237] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.099175 11891 rpcz_store.cc:276] 0113 08:59:13.006615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.209103 (+202488us) service_pool.cc:225] Handling call
0113 08:59:13.233779 (+ 24676us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:14.666923 (+1433144us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.098279 (+1431356us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540012f054. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.098539 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3145052416}
I20250113 08:59:16.100174 11893 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:16.101447 11888 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 2.618s user 0.011s sys 0.020s
I20250113 08:59:16.099536 14551 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:16.102085 11859 connection.cc:232] RPC call timeout handler was delayed by -3.0867802509999365s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
W20250113 08:59:16.103427 11888 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 2.620s user 0.011s sys 0.020s
I20250113 08:59:16.103899 11893 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20250113 08:59:16.103966 12006 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.645s user 0.020s sys 0.053s
I20250113 08:59:16.104317 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.105132 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:16.105597 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1642) took 5510 ms (client timeout 2153 ms). Trace:
W20250113 08:59:16.126375 12042 rpcz_store.cc:269] 0113 08:59:10.594637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:11.620931 (+1026294us) service_pool.cc:225] Handling call
0113 08:59:16.105198 (+4484267us) spinlock_profiling.cc:243] Waited 3.43 s on lock 0x7b54003688d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:16.105420 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7898017536}
W20250113 08:59:16.127327 12006 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.668s user 0.021s sys 0.053s
W20250113 08:59:16.127529 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 57528.4us
I20250113 08:59:16.127619 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.128355 12042 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.129559 11856 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.130900 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 568) took 6021 ms. Trace:
I20250113 08:59:16.131781 14604 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:16.132138 14473 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.691s user 0.004s sys 0.036s
W20250113 08:59:16.132504 14473 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.692s user 0.004s sys 0.036s
I20250113 08:59:16.132809 14522 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:16.132459 14604 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.133399 14522 raft_consensus.cc:695] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:16.133545 14580 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:16.134393 14588 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 2.392s user 0.018s sys 0.032s
W20250113 08:59:16.134838 14588 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 2.393s user 0.018s sys 0.032s
I20250113 08:59:16.134267 14580 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:16.134903 14604 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:16.135643 14647 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.136255 14647 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.136842 14580 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:16.160506 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.110869 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.111797 14636 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:16.117471 11822 debug-util.cc:398] Leaking SignalData structure 0x7b080026dbe0 after lost signal to thread 10702
W20250113 08:59:16.161615 11822 debug-util.cc:398] Leaking SignalData structure 0x7b080026dbc0 after lost signal to thread 10705
I20250113 08:59:16.161721 12037 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.164105 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.109802 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.166041 12036 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:16.159411 12685 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.167593 12724 raft_consensus.cc:1235] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:16.169833 12724 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20250113 08:59:16.117430 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 92735.5us
I20250113 08:59:16.134514 14522 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.118911 14481 consensus_queue.cc:1046] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:16.167740 12033 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:16.138808 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.131282 12726 rpcz_store.cc:276] 0113 08:59:10.109850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:12.265227 (+2155377us) service_pool.cc:225] Handling call
0113 08:59:15.729228 (+3464001us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.732228 (+ 3000us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.130691 (+398463us) spinlock_profiling.cc:243] Waited 3.42 s on lock 0x7b54006004d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:16.130838 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7876662144}
I20250113 08:59:16.139532 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.141635 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.131383 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.142066 14421 raft_consensus.cc:3049] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:16.094812 11968 rpcz_store.cc:276] 0113 08:59:12.753429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.668922 (+1915493us) service_pool.cc:225] Handling call
0113 08:59:14.669378 (+ 456us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.091875 (+1422497us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.093987 (+ 2112us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540029f254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.094222 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3268618496}
I20250113 08:59:16.142696 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.142983 11891 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:16.178582 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.143021 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.178974 11892 raft_consensus.cc:2405] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:16.135576 14604 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.144825 11884 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:16.146116 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:16.179847 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 102774us
I20250113 08:59:16.147039 14590 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.148563 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.150882 14341 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:16.152177 14560 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:16.147089 14595 log.cc:926] Time spent T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 3.513s user 0.008s sys 0.015s
I20250113 08:59:16.153371 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.157812 14461 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:16.160311 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.138397 14647 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.182904 11887 raft_consensus.cc:2405] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:16.180917 11885 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
W20250113 08:59:16.184795 11934 connection.cc:232] RPC call timeout handler was delayed by -1.5295358779999333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19720000000000001s for a response
I20250113 08:59:16.197059 11965 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:16.178376 14421 raft_consensus.cc:738] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:16.203028 11934 connection.cc:232] RPC call timeout handler was delayed by -1.1859529930000008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2667s for a response
I20250113 08:59:16.203192 12042 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
W20250113 08:59:16.203486 11934 connection.cc:232] RPC call timeout handler was delayed by -1.0985139849999541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
I20250113 08:59:16.204255 11934 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:16.204348 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.205818 12037 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:16.206693 11889 raft_consensus.cc:2405] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250113 08:59:16.207659 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.208014 12042 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:16.208812 11893 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:16.209656 12035 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:16.209439 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.210609 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.212337 12037 raft_consensus.cc:2388] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:16.212147 12007 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:16.211212 11887 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:16.216465 12033 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:16.222796 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.224176 12034 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:16.224298 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.228547 11890 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:16.229300 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.231441 11891 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.232196 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.232398 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.233315 12037 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:16.227447 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.231370 11856 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.238093 11885 raft_consensus.cc:2405] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250113 08:59:16.235155 11892 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:16.243058 11893 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250113 08:59:16.208870 14636 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.244436 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 528) took 2907 ms. Trace:
I20250113 08:59:16.238583 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.239634 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.245874 11889 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:16.234629 14604 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.248507 14601 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.248847 14604 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.249794 14421 consensus_queue.cc:260] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.250903 14421 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.253506 14636 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.255640 14636 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.228281 14460 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:16.259469 14460 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.259833 14460 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.261082 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.261844 12035 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.242892 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.261986 12685 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.227011 14647 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.260177 12036 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:16.272691 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.273427 11884 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:16.273586 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.270870 14291 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:16.276371 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.271207 11885 raft_consensus.cc:1235] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20250113 08:59:16.271426 14352 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:16.236096 14588 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:16.273756 12036 raft_consensus.cc:1235] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:16.270418 14393 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:16.244828 11888 rpcz_store.cc:276] 0113 08:59:13.336993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.363159 (+ 26166us) service_pool.cc:225] Handling call
0113 08:59:13.429953 (+ 66794us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:13.430242 (+ 289us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:16.136951 (+2706709us) log.cc:844] Serialized 14 byte log entry
0113 08:59:16.211842 (+ 74891us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:16.223812 (+ 11970us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:16.228113 (+ 4301us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.235462 (+ 7349us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:16.244383 (+ 8921us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61033,"prepare.queue_time_us":53864,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":2168,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":15479,"thread_start_us":2711957,"threads_started":2}
I20250113 08:59:16.230739 14580 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:59:16.279559 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 75383.4us
I20250113 08:59:16.287686 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:16.291903 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 61843.9us
W20250113 08:59:16.276933 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 51735.4us
I20250113 08:59:16.294207 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.292063 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.302937 12037 raft_consensus.cc:2388] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:16.274513 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.273911 14590 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.249171 14621 raft_consensus.cc:3049] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:16.300280 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:16.315871 11968 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.317811 14291 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.318950 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.320199 11890 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:16.321208 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.321343 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.321784 11890 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.274989 14481 consensus_queue.cc:1046] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20250113 08:59:16.322866 14481 raft_consensus.cc:3049] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:16.323186 14481 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:16.324071 14481 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.295581 14551 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:16.325024 12034 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250113 08:59:16.325695 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.325945 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.326169 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2204) took 3981 ms. Trace:
I20250113 08:59:16.326560 11888 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:16.326514 11884 raft_consensus.cc:2371] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5. Re-sending same reply.
I20250113 08:59:16.327589 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.297868 14393 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.328402 11884 raft_consensus.cc:2388] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:16.328526 14393 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.328910 12034 rpcz_store.cc:276] 0113 08:59:12.344288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.318047 (+3973759us) service_pool.cc:225] Handling call
0113 08:59:16.324710 (+ 6663us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.325783 (+ 1073us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.326054 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.331323 12034 raft_consensus.cc:1235] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250113 08:59:16.332087 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2205) took 3987 ms. Trace:
I20250113 08:59:16.332535 12034 rpcz_store.cc:276] 0113 08:59:12.344593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.330524 (+3985931us) service_pool.cc:225] Handling call
0113 08:59:16.330992 (+ 468us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.331793 (+ 801us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.332026 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.333490 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.335827 11885 raft_consensus.cc:2371] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4. Re-sending same reply.
I20250113 08:59:16.341775 12039 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:16.345635 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.346412 12722 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.350044 12718 raft_consensus.cc:1235] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250113 08:59:16.300782 14604 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.353636 11934 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:16.354553 14123 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.355149 14588 consensus_queue.cc:1046] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250113 08:59:16.356014 14588 consensus_queue.cc:1225] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 5f9c33777f6845c2848a2a6102a1c2ff log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250113 08:59:16.355944 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.356689 12685 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.357400 14609 raft_consensus.cc:3049] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:16.357589 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.357908 14609 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:16.359331 12041 raft_consensus.cc:1235] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250113 08:59:16.359594 14537 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:16.359091 14609 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.360436 12033 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:16.361069 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2199) took 4018 ms. Trace:
I20250113 08:59:16.361390 12033 rpcz_store.cc:276] 0113 08:59:12.342735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.262364 (+3919629us) service_pool.cc:225] Handling call
0113 08:59:16.360266 (+ 97902us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.360829 (+ 563us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.360983 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.362749 12033 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:16.363065 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2200) took 3967 ms. Trace:
I20250113 08:59:16.363286 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 531) took 3571 ms. Trace:
I20250113 08:59:16.363446 12036 rpcz_store.cc:276] 0113 08:59:12.343056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.273140 (+3930084us) service_pool.cc:225] Handling call
0113 08:59:16.273502 (+ 362us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.299841 (+ 26339us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.310480 (+ 10639us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.359956 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2211) took 3888 ms. Trace:
I20250113 08:59:16.365531 12041 rpcz_store.cc:276] 0113 08:59:12.471671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.358664 (+3886993us) service_pool.cc:225] Handling call
0113 08:59:16.359112 (+ 448us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.359732 (+ 620us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.359909 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.368536 12041 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:16.373829 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 533) took 3364 ms. Trace:
I20250113 08:59:16.374387 12041 rpcz_store.cc:276] 0113 08:59:13.009085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.367803 (+3358718us) service_pool.cc:225] Handling call
0113 08:59:16.368239 (+ 436us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.373501 (+ 5262us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.373760 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.376283 12041 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:16.377184 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2212) took 3342 ms. Trace:
W20250113 08:59:16.369529 11858 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.680s user 0.014s sys 0.016s
I20250113 08:59:16.369966 11856 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.378027 11959 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:16.378295 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 62942.8us
I20250113 08:59:16.379130 11856 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.380339 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.381429 14589 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:16.381870 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 444) took 3145 ms (client timeout 2096 ms). Trace:
I20250113 08:59:16.382139 14589 raft_consensus.cc:695] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:16.382269 11963 rpcz_store.cc:269] 0113 08:59:13.236124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:13.527767 (+291643us) service_pool.cc:225] Handling call
0113 08:59:16.381497 (+2853730us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54000514d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:16.381639 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":855185280}
I20250113 08:59:16.384330 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2173) took 3120 ms. Trace:
I20250113 08:59:16.383642 14589 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:16.369369 11822 debug-util.cc:398] Leaking SignalData structure 0x7b0800246340 after lost signal to thread 11788
I20250113 08:59:16.371444 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:16.385818 11822 debug-util.cc:398] Leaking SignalData structure 0x7b080031d120 after lost signal to thread 11825
I20250113 08:59:16.386097 14541 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:16.386523 14541 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.386909 14541 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:16.345703 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.387745 12724 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:16.353382 14604 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.387939 14590 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:16.390092 14604 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:16.386207 11822 debug-util.cc:398] Leaking SignalData structure 0x7b0800030020 after lost signal to thread 11839
I20250113 08:59:16.377010 14344 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:16.392480 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:16.393879 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 194462us
I20250113 08:59:16.366870 14609 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.366932 11931 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:16.394676 12041 rpcz_store.cc:276] 0113 08:59:13.035123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.375541 (+3340418us) service_pool.cc:225] Handling call
0113 08:59:16.376039 (+ 498us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.376741 (+ 702us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.377125 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.299906 14461 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:16.433037 11856 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:16.447660 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.758s user 0.014s sys 0.016s
I20250113 08:59:16.384660 11967 rpcz_store.cc:276] 0113 08:59:13.263588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.718978 (+1455390us) service_pool.cc:225] Handling call
0113 08:59:14.721634 (+ 2656us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.918171 (+1196537us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.384089 (+465918us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400381154. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.384276 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2528323968}
I20250113 08:59:16.405917 12042 raft_consensus.cc:2388] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:16.451705 12035 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:16.452131 14664 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:16.452669 12034 raft_consensus.cc:1235] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 6. Ops: [4.2-4.2]
I20250113 08:59:16.452787 14664 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.453390 14664 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.454437 12035 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:16.327888 11888 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250113 08:59:16.427472 12686 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:16.431306 12037 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:16.432464 12684 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:16.433189 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.433128 14563 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:16.434010 11886 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:16.365093 12036 raft_consensus.cc:1235] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:16.435783 11891 raft_consensus.cc:2405] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:16.438061 11968 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:16.418040 11890 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:16.363651 12033 rpcz_store.cc:276] 0113 08:59:12.791632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.362139 (+3570507us) service_pool.cc:225] Handling call
0113 08:59:16.362526 (+ 387us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.363072 (+ 546us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.363244 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.359968 14537 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.325695 14551 raft_consensus.cc:695] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:16.445935 14461 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.315539 14621 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:16.468560 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:16.481036 11890 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:16.312908 14647 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.381430 14547 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:16.446439 14481 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.468437 11893 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
W20250113 08:59:16.470361 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 65409.9us
I20250113 08:59:16.473070 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2202) took 4123 ms. Trace:
I20250113 08:59:16.459194 12041 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20250113 08:59:16.482038 11856 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:16.473378 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 534) took 3101 ms. Trace:
I20250113 08:59:16.482550 12007 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:16.487709 12037 rpcz_store.cc:276] 0113 08:59:12.343706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.396455 (+4052749us) service_pool.cc:225] Handling call
0113 08:59:16.431126 (+ 34671us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.467367 (+ 36241us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.467551 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.493203 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:16.454901 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 88033.8us
W20250113 08:59:16.475157 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 47806.7us
I20250113 08:59:16.488186 12035 rpcz_store.cc:276] 0113 08:59:13.363938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.453675 (+3089737us) service_pool.cc:225] Handling call
0113 08:59:16.454214 (+ 539us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.465503 (+ 11289us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.465810 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.398770 11934 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:16.487792 12041 pending_rounds.cc:77] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 6
I20250113 08:59:16.495971 12041 pending_rounds.cc:99] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20250113 08:59:16.496287 12041 raft_consensus.cc:2883] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:16.494379 11888 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 3.
I20250113 08:59:16.499555 11889 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:16.466017 14291 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:16.474715 14537 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.481525 14461 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.508131 11931 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:16.501368 12042 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20250113 08:59:16.511971 14460 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.499439 14497 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250113 08:59:16.512569 14393 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:16.465728 14541 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.513674 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.514844 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.517728 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2215) took 3335 ms. Trace:
I20250113 08:59:16.520737 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2213) took 3391 ms. Trace:
I20250113 08:59:16.523769 12042 rpcz_store.cc:276] 0113 08:59:13.182606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.451619 (+3269013us) service_pool.cc:225] Handling call
0113 08:59:16.489197 (+ 37578us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.517472 (+ 28275us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.517670 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.522027 11888 raft_consensus.cc:1235] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:16.521518 12033 raft_consensus.cc:1235] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250113 08:59:16.523895 12041 rpcz_store.cc:276] 0113 08:59:13.129896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.439496 (+3309600us) service_pool.cc:225] Handling call
0113 08:59:16.458752 (+ 19256us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.517861 (+ 59109us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.520665 (+ 2804us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.528530 11891 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:16.522941 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.531563 12035 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:16.500216 14604 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:16.534657 11889 raft_consensus.cc:1235] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:16.536423 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 180556us
I20250113 08:59:16.537566 14488 consensus_queue.cc:1046] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:16.538269 14393 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:16.538512 14541 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:16.539304 14291 consensus_queue.cc:1046] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:59:16.540035 14291 consensus_queue.cc:1225] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250113 08:59:16.540510 14291 raft_consensus.cc:3049] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:16.540809 14291 raft_consensus.cc:738] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:16.541239 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1658) took 4963 ms. Trace:
I20250113 08:59:16.541296 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:16.541689 14560 consensus_queue.cc:1046] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:59:16.542026 11963 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:16.542062 11967 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:11.019310 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250113 08:59:16.544179 11822 debug-util.cc:398] Leaking SignalData structure 0x7b0800054ce0 after lost signal to thread 11917
I20250113 08:59:16.503721 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 532) took 3708 ms. Trace:
I20250113 08:59:16.558777 12036 rpcz_store.cc:276] 0113 08:59:12.792015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.364415 (+3572400us) service_pool.cc:225] Handling call
0113 08:59:16.364769 (+ 354us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:16.472819 (+108050us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.499951 (+ 27132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.560022 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.560452 12036 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250113 08:59:16.560899 14460 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:16.561790 14560 raft_consensus.cc:3049] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:16.562217 14560 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:16.561828 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.563084 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:16.563751 11961 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.563891 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:16.564544 11963 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:16.565301 14604 raft_consensus.cc:695] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:16.547523 12686 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:16.566429 14604 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.568172 12035 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:16.568903 14473 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:59:16.569397 14150 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:16.570075 14150 raft_consensus.cc:695] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:16.570380 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33860 (request call id 849) took 6278 ms. Trace:
I20250113 08:59:16.571118 14194 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250113 08:59:16.571566 11822 debug-util.cc:398] Leaking SignalData structure 0x7b080040b3e0 after lost signal to thread 11992
W20250113 08:59:16.571928 11822 debug-util.cc:398] Leaking SignalData structure 0x7b080040b400 after lost signal to thread 11993
I20250113 08:59:16.572365 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2206) took 4227 ms. Trace:
W20250113 08:59:16.572595 11787 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44358 (request call id 346) took 15216 ms (client timeout 15000 ms). Trace:
I20250113 08:59:16.556713 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.557574 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:16.557873 14706 log.cc:926] Time spent T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250113 08:59:16.541438 14527 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20250113 08:59:16.562144 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:16.562768 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 526) took 4480 ms. Trace:
W20250113 08:59:16.567037 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 59973us
I20250113 08:59:16.538703 14488 raft_consensus.cc:3049] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:16.602569 12040 rpcz_store.cc:276] 0113 08:59:12.082191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.720728 (+2638537us) service_pool.cc:225] Handling call
0113 08:59:16.017555 (+1296827us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.023792 (+ 6237us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.562603 (+538811us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003bc250. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:16.562718 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2915962624}
W20250113 08:59:16.573091 11822 debug-util.cc:398] Leaking SignalData structure 0x7b080037b2c0 after lost signal to thread 12066
I20250113 08:59:16.604717 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:16.572726 12034 rpcz_store.cc:276] 0113 08:59:12.344885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.338683 (+3993798us) service_pool.cc:225] Handling call
0113 08:59:16.452445 (+113762us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.453054 (+ 609us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.572177 (+119123us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54002bd9d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:16.572314 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":260475136}
W20250113 08:59:16.574337 14708 log.cc:926] Time spent T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20250113 08:59:16.519713 14497 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.574715 14460 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250113 08:59:16.575129 14551 consensus_queue.cc:237] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.575462 14211 consensus_queue.cc:1046] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20250113 08:59:16.548820 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 116433us
I20250113 08:59:16.553545 14562 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:16.506484 14547 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:16.579826 12042 raft_consensus.cc:1235] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250113 08:59:16.631492 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.551970 11788 catalog_manager.cc:5526] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:59:16.541762 12038 rpcz_store.cc:276] 0113 08:59:11.577881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:14.710807 (+3132926us) service_pool.cc:225] Handling call
0113 08:59:14.728908 (+ 18101us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:15.772331 (+1043423us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.540985 (+768654us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54003bd654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.541188 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2397477632}
I20250113 08:59:16.562531 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.585335 11967 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:16.535423 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:16.633518 12037 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:16.488359 14580 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.570858 12035 rpcz_store.cc:276] 0113 08:59:10.291705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.568939 (+6277234us) service_pool.cc:225] Handling call
0113 08:59:16.570280 (+ 1341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:16.530304 14123 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:16.571821 14194 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.646896 12686 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:16.641588 11963 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:16.605723 11959 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:16.613749 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08003fec60 after lost signal to thread 12750
W20250113 08:59:16.651736 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08003fed20 after lost signal to thread 12751
I20250113 08:59:16.616482 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.620003 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.625788 11888 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:16.612501 14497 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Advancing to term 10
I20250113 08:59:16.610891 14344 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:16.604945 14488 raft_consensus.cc:738] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:16.651507 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:16.614197 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 77646.5us
I20250113 08:59:16.658735 11964 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:16.638626 14547 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.583760 14636 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20250113 08:59:16.601989 14406 consensus_queue.cc:1046] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
W20250113 08:59:16.652226 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08003d2120 after lost signal to thread 14333
I20250113 08:59:16.563163 14560 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.667014 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.668494 12724 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250113 08:59:16.685858 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:16.686278 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08002cc0e0 after lost signal to thread 14518
I20250113 08:59:16.685909 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.687417 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.571144 14150 consensus_queue.cc:237] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.683605 12722 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:16.675243 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.655469 14393 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:16.721374 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 81242.8us
I20250113 08:59:16.722965 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.626235 14621 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.659708 14460 raft_consensus.cc:3049] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:16.658675 14242 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250113 08:59:16.692577 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 119034us
W20250113 08:59:16.689728 12008 connection.cc:232] RPC call timeout handler was delayed by -1.7403022360000477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1633s for a response
I20250113 08:59:16.642513 14291 consensus_queue.cc:260] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:16.642019 14588 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250113 08:59:16.701035 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08003e59a0 after lost signal to thread 14583
I20250113 08:59:16.673195 14563 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:16.696702 14474 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:16.770273 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.711226 14407 raft_consensus.cc:2758] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250113 08:59:16.703938 14406 consensus_queue.cc:1225] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:16.696625 11788 catalog_manager.cc:5526] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 5, leader changed from 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 5 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:59:16.671480 14194 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:16.697202 12720 raft_consensus.cc:2388] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:16.704838 11932 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:16.696689 11839 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 79302a9220a149cb81cb2573664290a1, TS 6ea00811b7d44b49b872ac5c03b91487 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:16.580888 11787 rpcz_store.cc:269] 0113 08:59:01.354080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:01.354247 (+ 167us) service_pool.cc:225] Handling call
0113 08:59:16.570699 (+15216452us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:59:06.740940 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:59:07.020786 (+279846us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:59:07.020870 (+ 84us) write_op.cc:435] Acquired schema lock
0113 08:59:07.020893 (+ 23us) tablet.cc:659] Decoding operations
0113 08:59:07.021821 (+ 928us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:59:07.021928 (+ 107us) write_op.cc:641] Partition lock acquired for write op
0113 08:59:07.021975 (+ 47us) tablet.cc:682] Acquiring locks for 4 operations
0113 08:59:07.022618 (+ 643us) tablet.cc:698] Row locks acquired
0113 08:59:07.022639 (+ 21us) write_op.cc:260] PREPARE: finished
0113 08:59:07.022884 (+ 245us) write_op.cc:270] Start()
0113 08:59:07.023046 (+ 162us) write_op.cc:276] Timestamp: P: 1736758747022775 usec, L: 0
0113 08:59:07.023070 (+ 24us) op_driver.cc:348] REPLICATION: starting
0113 08:59:07.023969 (+ 899us) log.cc:844] Serialized 1698 byte log entry
0113 08:59:08.847581 (+1823612us) op_driver.cc:464] REPLICATION: finished
0113 08:59:09.032651 (+185070us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3336b310e 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336d08c2 00007fc33a01cd20
0113 08:59:10.017757 (+985106us) write_op.cc:301] APPLY: starting
0113 08:59:10.039378 (+ 21621us) tablet.cc:1364] starting BulkCheckPresence
0113 08:59:10.112902 (+ 73524us) tablet.cc:1367] finished BulkCheckPresence
0113 08:59:10.140516 (+ 27614us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:59:10.282471 (+141955us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:59:10.282735 (+ 264us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0113 08:59:10.282775 (+ 40us) write_op.cc:312] APPLY: finished
0113 08:59:10.283135 (+ 360us) log.cc:844] Serialized 41 byte log entry
0113 08:59:10.641047 (+357912us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:10.649386 (+ 8339us) write_op.cc:454] Released schema lock
0113 08:59:10.693429 (+ 44043us) write_op.cc:341] FINISH: Updating metrics
0113 08:59:11.211101 (+517672us) spinlock_profiling.cc:243] Waited 51.9 ms on lock 0x7b1800145d20. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d1204 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:59:11.211254 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:11.211293 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":529190,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":902707,"apply.run_cpu_time_us":29180,"apply.run_wall_time_us":814011,"num_ops":4,"prepare.queue_time_us":608067,"prepare.run_cpu_time_us":16859,"prepare.run_wall_time_us":820725,"raft.queue_time_us":339763,"raft.run_cpu_time_us":13277,"raft.run_wall_time_us":318267,"replication_time_us":1824329,"spinlock_wait_cycles":146823296,"thread_start_us":1701154,"threads_started":5,"wal-append.queue_time_us":1593533,"wal-append.run_cpu_time_us":19562,"wal-append.run_wall_time_us":2988971}]]}
I20250113 08:59:16.683291 14527 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:16.718809 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 50346.6us
I20250113 08:59:16.669548 11959 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:16.728154 11963 raft_consensus.cc:2388] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:16.722318 12726 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:16.730726 12725 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:16.683379 14211 consensus_queue.cc:1046] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
W20250113 08:59:16.740242 11822 debug-util.cc:398] Leaking SignalData structure 0x7b0800388700 after lost signal to thread 14592
I20250113 08:59:16.686514 14123 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20250113 08:59:16.773360 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.906155 11887 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:16.773386 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:16.906832 12042 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:16.698227 14645 consensus_queue.cc:1035] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20250113 08:59:16.766902 11858 connection.cc:232] RPC call timeout handler was delayed by -1.4778454119999651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20250113 08:59:16.907830 11858 connection.cc:232] RPC call timeout handler was delayed by -1.2636062249999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
I20250113 08:59:16.798323 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:16.803547 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68761.5us
I20250113 08:59:16.908716 12035 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:59:16.771179 14705 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20250113 08:59:16.893633 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08003f0ca0 after lost signal to thread 14593
W20250113 08:59:16.894240 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 79302a9220a149cb81cb2573664290a1, TS 6ea00811b7d44b49b872ac5c03b91487 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:16.927121 11822 debug-util.cc:398] Leaking SignalData structure 0x7b08001c7480 after lost signal to thread 14596
I20250113 08:59:16.731940 14460 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:16.897174 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 192421us
I20250113 08:59:16.773957 14647 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:16.924211 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:16.773803 14560 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.834508 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:16.924021 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1665) took 3417 ms. Trace:
I20250113 08:59:16.967231 12033 rpcz_store.cc:276] 0113 08:59:13.429615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.468870 (+3039255us) service_pool.cc:225] Handling call
0113 08:59:16.508258 (+ 39388us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.535747 (+ 27489us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.846593 (+310846us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b54003692d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:16.846861 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29687680}
I20250113 08:59:16.847607 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:16.739596 14242 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:16.740038 14352 consensus_queue.cc:1046] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
I20250113 08:59:16.965674 12723 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:16.718279 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:16.970482 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 60512us
I20250113 08:59:16.925055 11788 catalog_manager.cc:5526] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 3 to 4, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 4 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250113 08:59:16.972491 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 471236us
I20250113 08:59:16.946242 12007 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:16.766434 14561 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.175s
I20250113 08:59:16.990281 12686 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:16.992888 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.993980 11963 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:16.829831 14589 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:16.991855 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 45706.2us
I20250113 08:59:16.995417 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:16.989461 14392 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:17.003448 12007 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:16.972626 14242 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:59:17.006039 14709 log.cc:926] Time spent T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250113 08:59:17.005177 11788 catalog_manager.cc:5526] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 3 to 4, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 4 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:59:17.006502 14580 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:17.006628 14392 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.006577 14561 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250113 08:59:16.995651 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.741029 14393 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.048646 12725 raft_consensus.cc:3049] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
W20250113 08:59:17.023021 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 71281.9us
W20250113 08:59:16.995296 14595 log.cc:926] Time spent T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20250113 08:59:17.008702 11931 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:17.000805 14701 log.cc:926] Time spent T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20250113 08:59:17.012724 11932 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:17.035555 12726 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:17.027627 11893 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:16.996002 14588 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:16.999724 14473 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:17.049223 12725 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:17.061759 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:17.062525 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 128942us
W20250113 08:59:17.064267 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 562233us
I20250113 08:59:17.021404 14352 consensus_queue.cc:1225] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:17.050089 14590 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:17.084849 11788 catalog_manager.cc:5526] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 3 to 4, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 4 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:59:17.027278 14406 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:16.925840 14488 consensus_queue.cc:260] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.068111 12684 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:16.946195 14527 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:17.021831 14647 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:17.068673 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1669) took 1195 ms. Trace:
I20250113 08:59:17.126873 12039 rpcz_store.cc:276] 0113 08:59:15.873728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.008494 (+134766us) service_pool.cc:225] Handling call
0113 08:59:17.068449 (+1059955us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54003697d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:17.068596 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":746283776}
I20250113 08:59:17.092132 12042 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:17.098305 14392 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:17.098394 14393 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:17.099012 12686 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.113601 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:16.952483 14474 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.108392 14194 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:17.154155 11893 raft_consensus.cc:2405] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:17.142434 12723 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
W20250113 08:59:17.108031 14461 consensus_meta.cc:287] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.561s user 0.008s sys 0.002s
I20250113 08:59:17.177525 14461 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.179723 14461 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:17.181672 14588 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:17.182235 14588 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:17.183120 14588 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.185732 12725 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.186611 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.186903 12725 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.188050 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.188618 11893 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:17.189373 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.190615 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:17.120234 14539 consensus_queue.cc:1046] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20250113 08:59:17.193162 14539 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:17.193603 14539 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:17.195111 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.194509 14539 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.158633 14474 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:17.159395 14527 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:17.160526 14242 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.161861 14406 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:17.052558 14589 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:17.162050 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 105760us
I20250113 08:59:17.162707 14715 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250113 08:59:17.169716 14713 log.cc:926] Time spent T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20250113 08:59:17.212394 14585 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:17.214524 14585 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.214928 14585 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:17.214758 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:17.215446 12684 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.216638 14406 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:17.195861 11892 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:17.221589 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:17.157804 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 67464.4us
I20250113 08:59:17.154130 11788 catalog_manager.cc:5526] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 5 to 6, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 6 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:59:17.200222 11885 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:17.091189 14473 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.205863 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.101212 14497 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:17.210628 11961 consensus_meta.cc:287] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.646s user 0.009s sys 0.001s
I20250113 08:59:17.203936 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.101486 14460 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.126657 14647 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.241384 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.242913 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.224607 11887 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:17.258102 12041 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.042s sys 0.040s
I20250113 08:59:17.258294 11893 raft_consensus.cc:2388] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:17.224995 11886 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:17.208750 12726 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250113 08:59:17.259557 12041 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.042s sys 0.040s
I20250113 08:59:17.260661 12685 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:17.261612 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 493553us
I20250113 08:59:17.250748 11889 raft_consensus.cc:1270] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:17.259310 11961 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:17.228024 11788 catalog_manager.cc:5526] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 5, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 5 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250113 08:59:17.262403 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 488824us
I20250113 08:59:17.268633 11892 raft_consensus.cc:2405] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250113 08:59:17.211707 14710 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.404s user 0.000s sys 0.001s
I20250113 08:59:17.276647 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:17.278420 11887 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:17.244218 14712 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.337s user 0.000s sys 0.001s
W20250113 08:59:17.280751 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.010s sys 0.026s
I20250113 08:59:17.298619 12685 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:17.299180 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.676s user 0.010s sys 0.026s
W20250113 08:59:17.299638 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 676368us
I20250113 08:59:17.306250 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.232357 14242 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:17.227701 14473 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.307641 11934 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:17.312459 11885 raft_consensus.cc:1235] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:17.312727 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:17.333871 14537 consensus_meta.cc:287] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.810s user 0.010s sys 0.000s
I20250113 08:59:17.244218 14563 consensus_queue.cc:1046] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20250113 08:59:17.306643 14636 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20250113 08:59:17.334196 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.335197 14211 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:17.335817 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.336199 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.336716 11886 raft_consensus.cc:2405] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:17.337291 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.337724 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.338541 11886 raft_consensus.cc:2405] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:17.337072 12718 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:17.336807 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.339591 11886 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250113 08:59:17.319936 14547 consensus_meta.cc:287] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.649s user 0.008s sys 0.001s
I20250113 08:59:17.340387 14547 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.314846 11788 catalog_manager.cc:5526] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff reported cstate change: term changed from 2 to 3, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 3 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250113 08:59:17.335813 14636 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.343487 12686 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:17.326261 14609 consensus_meta.cc:287] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.897s user 0.011s sys 0.000s
I20250113 08:59:17.310109 14461 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:17.334568 14563 consensus_queue.cc:1225] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:17.345667 14461 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.346153 14563 raft_consensus.cc:2758] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250113 08:59:17.346259 14461 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:17.346180 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:17.334194 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:17.348768 11885 raft_consensus.cc:2405] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250113 08:59:17.349457 14610 raft_consensus.cc:3049] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:17.349879 14610 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:17.350273 11965 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:17.350816 14610 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.352137 14610 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.371946 12725 raft_consensus.cc:1270] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250113 08:59:17.365182 11887 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
W20250113 08:59:17.379112 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 74787.8us
I20250113 08:59:17.380589 11856 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:17.379187 12725 pending_rounds.cc:77] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting all ops after (but not including) 6
I20250113 08:59:17.381588 12725 pending_rounds.cc:99] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250113 08:59:17.381906 12725 raft_consensus.cc:2883] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:17.383688 14562 consensus_queue.cc:1046] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:17.384971 14497 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 10 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:17.389550 12008 leader_election.cc:336] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.724s (SENT)
I20250113 08:59:17.390087 12008 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:17.390885 14467 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:17.392524 12008 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.393492 14474 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:17.392817 14467 consensus_queue.cc:1046] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.002s
I20250113 08:59:17.383715 14563 consensus_queue.cc:1046] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20250113 08:59:17.395206 14467 raft_consensus.cc:3049] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:17.395637 14467 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:17.396725 14467 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.398515 14467 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.400437 14647 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:17.403497 14636 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:17.369797 14344 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20250113 08:59:17.410743 14664 consensus_meta.cc:287] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.957s user 0.011s sys 0.000s
I20250113 08:59:17.407323 14589 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:17.407716 14710 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250113 08:59:17.411219 14664 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.345358 14718 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:17.413762 14718 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.415856 14718 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:17.417611 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:17.408087 14461 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.419533 12724 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:17.310248 14455 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:17.425756 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45263.9us
I20250113 08:59:17.433077 11891 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:17.442482 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:17.443908 11859 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:17.445639 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:17.450865 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.449162 11858 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:17.464646 12042 raft_consensus.cc:2388] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:17.466483 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:17.459956 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 409826us
I20250113 08:59:17.469492 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.468276 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:17.471376 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 105110us
I20250113 08:59:17.469055 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:17.472183 12684 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.472613 12723 raft_consensus.cc:2388] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:17.334353 14537 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.445156 14718 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:17.476791 12726 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:17.476028 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.481071 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.482446 12036 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:17.483219 12717 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:17.483848 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 815) took 1140 ms. Trace:
I20250113 08:59:17.484233 12717 rpcz_store.cc:276] 0113 08:59:16.343124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.343275 (+ 151us) service_pool.cc:225] Handling call
0113 08:59:17.214575 (+871300us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:17.483775 (+269200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6649216}
I20250113 08:59:17.484787 14406 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:17.448945 14636 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:17.485507 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 93114.2us
I20250113 08:59:17.485780 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.485589 14636 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.487190 12008 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.487869 14636 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:17.488590 14474 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:17.489095 14474 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.489526 14474 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.459221 14563 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:17.490298 14563 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.492449 14563 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:17.493841 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:17.443620 14725 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20250113 08:59:17.494467 12034 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Advancing to term 10
I20250113 08:59:17.495015 14537 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:17.495393 12033 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:17.476944 12725 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:17.497979 12039 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250113 08:59:17.371169 14609 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20250113 08:59:17.501653 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.367444 14715 consensus_queue.cc:1046] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20250113 08:59:17.502985 14718 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:17.503584 14718 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:17.503769 14715 raft_consensus.cc:3049] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:17.504287 14715 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:17.460844 14664 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:17.459185 14590 raft_consensus.cc:695] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:17.489619 14585 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.506428 14590 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:17.507145 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2200) took 1115 ms. Trace:
I20250113 08:59:17.507373 14585 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:17.489557 14318 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:17.488533 14724 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:17.507681 11962 rpcz_store.cc:276] 0113 08:59:16.391892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.392504 (+ 612us) service_pool.cc:225] Handling call
0113 08:59:16.392829 (+ 325us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.393219 (+ 390us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:16.393266 (+ 47us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:16.393736 (+ 470us) log.cc:844] Serialized 26 byte log entry
0113 08:59:16.610357 (+216621us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:16.610391 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:16.610461 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:17.506877 (+896416us) raft_consensus.cc:1680] finished
0113 08:59:17.506903 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:17.507074 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11174,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":34113,"thread_start_us":216166,"threads_started":1,"wal-append.queue_time_us":323484}
I20250113 08:59:17.501968 12039 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:17.514870 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:17.519850 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 546975us
I20250113 08:59:17.542233 11932 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:17.497581 14474 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.529865 12034 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 10.
I20250113 08:59:17.580852 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:17.581372 12033 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:17.581622 12034 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:17.543239 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:17.582867 11890 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:17.509788 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:17.583101 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 93960.6us
I20250113 08:59:17.584437 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:17.585968 11891 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:17.587226 12685 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:17.587831 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 115790us
W20250113 08:59:17.593569 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.002s sys 0.038s
W20250113 08:59:17.594017 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.532s user 0.002s sys 0.038s
W20250113 08:59:17.594329 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 595091us
I20250113 08:59:17.606268 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:17.607062 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:17.607738 11887 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:17.608029 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 183210us
I20250113 08:59:17.617575 12042 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:17.627887 12033 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250113 08:59:17.629529 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 45185.6us
I20250113 08:59:17.630076 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:17.640340 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 56326.3us
I20250113 08:59:17.509140 14461 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:17.653927 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 65196.4us
I20250113 08:59:17.634230 12039 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:17.656309 11963 raft_consensus.cc:1235] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:17.654889 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:17.666219 11966 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:17.671175 12007 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:17.614805 14604 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.013s sys 0.026s
I20250113 08:59:17.671064 11887 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:17.628913 14585 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:59:17.641330 14406 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250113 08:59:17.697985 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 103060us
I20250113 08:59:17.698510 11932 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.699024 11934 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.676666 11890 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:17.655565 14537 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:17.703703 11856 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:17.640017 14318 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:17.674233 14604 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.990s user 0.015s sys 0.028s
I20250113 08:59:17.512921 14473 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.714537 14474 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:17.715230 14723 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:17.697463 12036 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:17.716264 14318 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.708235 11931 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:17.709216 14589 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:17.709885 14731 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:17.674400 14242 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.036s
I20250113 08:59:17.729630 12008 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:17.714692 14636 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:17.661103 14724 raft_consensus.cc:695] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:17.708343 12034 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:59:17.722304 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 79285.1us
W20250113 08:59:17.743836 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45111.2us
I20250113 08:59:17.744884 11931 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:17.731762 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 261053us
I20250113 08:59:17.678038 14585 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:17.679930 14664 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20250113 08:59:17.703359 14406 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.629165 14150 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20250113 08:59:17.744645 11859 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:17.727517 14589 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.515184 14715 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.747586 12039 raft_consensus.cc:1270] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250113 08:59:17.720805 14344 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:17.715333 14473 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:17.791741 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 46972us
I20250113 08:59:17.757187 12684 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.758255 11856 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:17.758867 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 326743us
I20250113 08:59:17.671661 14551 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
W20250113 08:59:17.747196 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 48797.3us
I20250113 08:59:17.798370 11932 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:17.798431 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:17.763075 12034 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 6. Ops: []
I20250113 08:59:17.741406 14455 raft_consensus.cc:695] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:17.700819 14527 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20250113 08:59:17.793102 11931 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.744061 14467 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:17.727779 14731 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.760159 14585 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:59:17.761363 14406 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:17.813596 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 142500us
W20250113 08:59:17.806294 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 59415.3us
I20250113 08:59:17.830087 12042 raft_consensus.cc:1235] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:17.772043 14589 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.836123 12725 raft_consensus.cc:1270] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:17.836336 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.841274 12726 raft_consensus.cc:2405] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250113 08:59:17.836766 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 92210us
I20250113 08:59:17.770783 14723 raft_consensus.cc:2743] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:17.810653 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:17.850903 12038 raft_consensus.cc:2388] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:17.854892 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:17.855791 11966 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.799589 14636 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:17.806846 14727 raft_consensus.cc:2758] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250113 08:59:17.803738 14610 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 FOLLOWER]: Leader election won for term 10
W20250113 08:59:17.862061 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 61769.5us
I20250113 08:59:17.863041 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.863981 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2229) took 1172 ms. Trace:
I20250113 08:59:17.864374 12038 rpcz_store.cc:276] 0113 08:59:16.691815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.810352 (+1118537us) service_pool.cc:225] Handling call
0113 08:59:17.863919 (+ 53567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:17.826354 14562 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:17.787714 14497 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20250113 08:59:17.829216 14467 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:17.877707 14562 raft_consensus.cc:695] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:17.810621 14150 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.058s
I20250113 08:59:17.877971 14455 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.878892 14562 consensus_queue.cc:237] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:17.803313 14604 consensus_queue.cc:1035] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20250113 08:59:17.862131 12718 raft_consensus.cc:2405] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20250113 08:59:17.829499 14731 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:17.881812 14728 log.cc:926] Time spent T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
W20250113 08:59:17.883563 14734 log.cc:926] Time spent T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250113 08:59:17.885175 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:17.886162 14730 log.cc:926] Time spent T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20250113 08:59:17.886121 12717 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:17.890695 11959 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:17.891589 11966 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:17.798928 14474 consensus_queue.cc:1035] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20250113 08:59:17.902112 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:17.920878 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 127858us
I20250113 08:59:17.921978 11931 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.925113 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1666) took 4495 ms. Trace:
I20250113 08:59:17.925621 12041 rpcz_store.cc:276] 0113 08:59:13.429931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.525023 (+3095092us) service_pool.cc:225] Handling call
0113 08:59:16.576357 (+ 51334us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.651439 (+ 75082us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:16.651466 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:16.658588 (+ 7122us) log.cc:844] Serialized 26 byte log entry
0113 08:59:17.261642 (+603054us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:17.699731 (+438089us) log.cc:844] Serialized 14 byte log entry
0113 08:59:17.924727 (+224996us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:17.924818 (+ 91us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:17.924872 (+ 54us) raft_consensus.cc:1680] finished
0113 08:59:17.924886 (+ 14us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:17.925049 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30693,"prepare.queue_time_us":588605,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":32468,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":11501,"spinlock_wait_cycles":114944,"thread_start_us":1154376,"threads_started":3,"wal-append.queue_time_us":691943}
I20250113 08:59:17.927965 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 551) took 1164 ms. Trace:
W20250113 08:59:17.928959 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 66130.1us
W20250113 08:59:17.929126 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 130827us
I20250113 08:59:17.929605 12007 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:17.929893 11932 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.887434 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:17.876695 14645 consensus_queue.cc:1035] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20250113 08:59:17.914366 14537 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:17.946101 12037 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250113 08:59:17.887924 14467 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:17.947921 14738 log.cc:926] Time spent T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250113 08:59:17.952004 12721 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:17.952788 14736 raft_consensus.cc:2758] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250113 08:59:17.881806 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.797816 11963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c"
dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:46296
I20250113 08:59:17.953897 11963 raft_consensus.cc:604] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:17.954265 11963 raft_consensus.cc:612] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250113 08:59:17.954600 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:17.955972 12042 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20250113 08:59:17.945730 14537 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:17.877812 14724 consensus_queue.cc:237] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:17.903267 14705 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250113 08:59:17.955282 12720 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:17.965202 11858 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:17.969807 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 47989.4us
I20250113 08:59:17.970592 11931 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:17.961138 12725 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:17.962806 12685 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:17.928488 11884 rpcz_store.cc:276] 0113 08:59:16.737706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.737872 (+ 166us) service_pool.cc:225] Handling call
0113 08:59:16.804434 (+ 66562us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:17.267719 (+463285us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:17.267760 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:17.306348 (+ 38588us) log.cc:844] Serialized 26 byte log entry
0113 08:59:17.654515 (+348167us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:17.654552 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:17.654618 (+ 66us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:17.888651 (+234033us) raft_consensus.cc:1680] finished
0113 08:59:17.888677 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:17.901970 (+ 13293us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":214553,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":24830,"thread_start_us":680751,"threads_started":2,"wal-append.queue_time_us":550896}
I20250113 08:59:17.869655 14723 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:17.973232 12685 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:17.866286 14242 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.092s
I20250113 08:59:17.975546 11931 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:17.973927 14407 consensus_queue.cc:1046] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.003s
W20250113 08:59:17.976672 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 133887us
W20250113 08:59:17.967535 14712 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250113 08:59:17.977501 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250113 08:59:17.977779 14727 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:17.941910 14590 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:17.973990 14541 raft_consensus.cc:3049] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:17.980687 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:17.986464 11859 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:17.928279 14610 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:17.957863 14636 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:17.963640 14749 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:18.006640 14740 log.cc:926] Time spent T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20250113 08:59:17.976778 14723 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.007700 11932 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:18.007853 14609 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250113 08:59:17.979384 14711 log.cc:926] Time spent T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 1.263s user 0.004s sys 0.000s
I20250113 08:59:18.001010 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250113 08:59:17.979607 14123 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:17.940809 14580 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.058s
I20250113 08:59:17.988216 14664 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250113 08:59:18.027274 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 62175.8us
I20250113 08:59:18.097653 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 549) took 1628 ms. Trace:
W20250113 08:59:18.089870 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 82235.5us
I20250113 08:59:18.098066 11888 rpcz_store.cc:276] 0113 08:59:16.469154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.525026 (+ 55872us) service_pool.cc:225] Handling call
0113 08:59:16.538121 (+ 13095us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:16.655101 (+116980us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:18.097401 (+1442300us) spinlock_profiling.cc:243] Waited 87.3 ms on lock 0x7b54001702d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:18.097595 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200747264}
I20250113 08:59:18.013720 14541 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:18.020416 14551 raft_consensus.cc:2743] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250113 08:59:18.028220 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 221307us
W20250113 08:59:18.100574 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 127406us
I20250113 08:59:18.098439 11932 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:18.094309 12725 raft_consensus.cc:1235] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250113 08:59:18.026593 14590 raft_consensus.cc:695] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:18.185894 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:18.094419 14718 consensus_meta.cc:287] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.590s user 0.010s sys 0.001s
I20250113 08:59:18.186633 12722 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:59:18.192986 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 735) took 1486 ms (client timeout 1900 ms). Trace:
I20250113 08:59:18.193862 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250113 08:59:18.194085 12722 rpcz_store.cc:269] 0113 08:59:16.707144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.715761 (+ 8617us) service_pool.cc:225] Handling call
0113 08:59:18.185695 (+1469934us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:18.192919 (+ 7224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5023104}
W20250113 08:59:18.195348 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 96998.8us
I20250113 08:59:18.097112 14727 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:18.096503 14664 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:18.029526 14318 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.163s
I20250113 08:59:18.093288 14636 raft_consensus.cc:695] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:18.026301 14129 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.165s
W20250113 08:59:18.213150 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 115430us
I20250113 08:59:18.230054 11858 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:18.235217 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 248879us
I20250113 08:59:18.245452 14636 consensus_queue.cc:237] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:18.247105 14745 log.cc:926] Time spent T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
W20250113 08:59:18.250545 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 54471.4us
I20250113 08:59:18.251510 11932 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:18.260497 11858 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:18.223201 14664 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:18.212280 14497 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:18.245867 14759 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:18.272905 12008 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:18.248512 14767 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:18.277949 14751 consensus_queue.cc:1046] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:18.278263 14744 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:18.264977 14609 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:18.278774 14760 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:18.281823 11932 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:18.227105 14727 raft_consensus.cc:695] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:18.284232 11858 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:18.202714 14718 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.279359 14732 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:18.035880 14749 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:18.333909 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 53304us
I20250113 08:59:18.336352 11859 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:18.337625 14497 raft_consensus.cc:695] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:18.338865 14497 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:18.341670 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 68840.9us
I20250113 08:59:18.285527 14242 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:18.298736 14344 consensus_queue.cc:1046] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20250113 08:59:18.354321 11887 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:18.306079 14609 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:18.344516 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 62387.4us
I20250113 08:59:18.327503 14537 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:18.353477 14129 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:18.343621 14767 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:18.386543 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:18.387295 14767 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:18.387898 14767 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:18.390290 14242 raft_consensus.cc:695] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:18.390682 14291 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:18.391270 14242 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.266470 14610 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:18.379252 14585 consensus_meta.cc:287] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.520s user 0.009s sys 0.000s
I20250113 08:59:18.403142 11859 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:18.376140 14724 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 LEADER]: Stepping down as leader of term 9
I20250113 08:59:18.413635 14724 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:18.414095 14522 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:18.414741 14522 raft_consensus.cc:695] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:18.414934 14421 raft_consensus.cc:3049] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:18.415469 14421 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:18.415753 14522 consensus_queue.cc:237] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.375062 14609 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:18.299160 14759 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.406198 14780 consensus_queue.cc:1035] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
W20250113 08:59:18.446584 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 43560.5us
I20250113 08:59:18.447793 11859 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:18.379822 14580 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:18.400104 14344 consensus_queue.cc:1225] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:18.298264 14736 consensus_queue.cc:1046] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20250113 08:59:18.388268 14767 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:18.416574 14421 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.461863 14421 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:18.326515 14590 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.467118 11859 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:18.469285 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 187567us
I20250113 08:59:18.455575 14580 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:18.520824 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 236380us
I20250113 08:59:18.435063 14585 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:18.396435 14760 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.103s
W20250113 08:59:18.557582 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 140531us
I20250113 08:59:18.522625 11858 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:18.525081 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 336573us
I20250113 08:59:18.468681 14767 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:18.458871 14787 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.575577 11966 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:18.587244 11961 raft_consensus.cc:2405] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250113 08:59:18.519999 14580 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:18.523545 14736 consensus_queue.cc:1225] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:18.524084 14621 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:18.579465 14749 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:18.519579 14731 consensus_meta.cc:287] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.637s user 0.007s sys 0.004s
I20250113 08:59:18.523896 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:18.580577 14406 consensus_meta.cc:287] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.722s user 0.011s sys 0.000s
I20250113 08:59:18.519390 14744 raft_consensus.cc:695] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:18.605742 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 83218.7us
W20250113 08:59:18.611621 12720 consensus_meta.cc:287] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.603s user 0.010s sys 0.000s
I20250113 08:59:18.603384 14718 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:18.615794 14759 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:18.471712 14723 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:18.622308 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 731) took 1966 ms. Trace:
I20250113 08:59:18.541055 14792 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.607266 14752 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:18.622738 14406 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.666574 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:18.654917 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:18.668088 11960 raft_consensus.cc:2388] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:18.434688 14727 consensus_queue.cc:237] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:18.659457 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 76263.5us
W20250113 08:59:18.641490 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 75964.6us
I20250113 08:59:18.664472 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:18.673866 11963 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:18.658311 14723 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:18.656790 14749 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:18.626516 12720 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:18.658182 12719 rpcz_store.cc:276] 0113 08:59:16.655960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.663026 (+ 7066us) service_pool.cc:225] Handling call
0113 08:59:18.521382 (+1858356us) spinlock_profiling.cc:243] Waited 6.62 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc33369360e 0000000000353db9 00007fc33b4f2a50 00007fc33b5b8814 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:18.622218 (+100836us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15229824}
I20250113 08:59:18.685979 12723 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:18.591763 14770 consensus_queue.cc:1046] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.300s
W20250113 08:59:18.687687 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 220678us
I20250113 08:59:18.665872 14718 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:18.707274 12037 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:18.718053 11859 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:18.721498 12008 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:18.584158 14787 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.626371 14731 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.687772 14769 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.686986 14749 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:18.751706 14749 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:18.751960 14781 log.cc:926] Time spent T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20250113 08:59:18.739961 12006 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:18.751984 14791 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:18.739647 14467 consensus_meta.cc:287] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.793s user 0.006s sys 0.005s
I20250113 08:59:18.739774 14789 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.753631 14467 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.750061 14776 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.741901 14807 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250113 08:59:18.750108 14801 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:18.750627 14757 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:18.686062 14779 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:18.789783 11858 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:18.768527 11859 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:18.769682 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 52133.5us
I20250113 08:59:18.658507 14792 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:18.731917 14777 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:18.775756 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 96615.8us
I20250113 08:59:18.775760 12035 raft_consensus.cc:2388] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:18.742050 12719 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEARNER]: Advancing to term 6
I20250113 08:59:18.791391 11963 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:18.796681 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:18.830562 11965 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:18.834441 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:18.834453 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:18.774736 14801 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:18.837953 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:18.754582 14789 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.836676 11884 raft_consensus.cc:2371] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6. Re-sending same reply.
I20250113 08:59:18.833132 11960 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:18.795755 14723 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.846350 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:18.674520 14752 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.862198 11858 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:18.805609 14585 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:18.873317 12034 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 6. Ops: []
I20250113 08:59:18.838929 14757 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.741513 14744 consensus_queue.cc:237] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.800240 14759 consensus_queue.cc:1046] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
W20250113 08:59:18.891331 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.001s sys 0.022s
I20250113 08:59:18.851342 14801 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:18.910998 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.003s sys 0.023s
I20250113 08:59:18.870178 14522 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:18.922191 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:18.759836 14769 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.903620 14736 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.936519 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:18.912135 14590 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:18.863720 14723 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:18.860468 14795 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.086s
I20250113 08:59:18.927234 14406 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:18.937222 12039 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:18.938853 14798 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.940737 14736 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.799394 14779 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:18.831442 14777 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:18.945327 14778 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.945536 14802 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:18.778590 14776 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.933467 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:18.963135 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 132807us
I20250113 08:59:18.930999 14759 consensus_queue.cc:1225] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250113 08:59:18.990358 14547 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:18.933322 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2232) took 1434 ms. Trace:
I20250113 08:59:18.992242 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:18.993698 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:18.994442 11963 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:18.994076 14724 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.995525 14724 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Advancing to term 10
I20250113 08:59:18.996109 14194 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.996446 14795 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:18.996796 14194 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:18.999814 14194 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:19.002219 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 599) took 1665 ms. Trace:
I20250113 08:59:19.002707 12724 rpcz_store.cc:276] 0113 08:59:17.336399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.338345 (+ 1946us) service_pool.cc:225] Handling call
0113 08:59:17.338654 (+ 309us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:17.420041 (+ 81387us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:19.001884 (+1581843us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x7b54006e05d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:19.002099 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185381248}
W20250113 08:59:19.008214 14772 log.cc:926] Time spent T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.275s user 0.000s sys 0.001s
W20250113 08:59:19.008608 14806 log.cc:926] Time spent T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
W20250113 08:59:18.947075 12717 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.010s sys 0.010s
I20250113 08:59:18.986011 12037 raft_consensus.cc:2388] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:18.991451 14211 raft_consensus.cc:3049] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:19.012526 11962 raft_consensus.cc:3049] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:19.012912 11962 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:18.949705 14467 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:18.966362 14522 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.106s
I20250113 08:59:18.904330 14757 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:18.991918 12036 rpcz_store.cc:276] 0113 08:59:17.496873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.499003 (+ 2130us) service_pool.cc:225] Handling call
0113 08:59:17.499361 (+ 358us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:17.715950 (+216589us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:18.905189 (+1189239us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54001fdbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:18.931181 (+ 25992us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":361876224}
W20250113 08:59:19.021643 14773 log.cc:926] Time spent T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.350s user 0.000s sys 0.001s
W20250113 08:59:19.280809 14580 consensus_meta.cc:287] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.674s user 0.010s sys 0.002s
I20250113 08:59:19.229110 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 552) took 1474 ms. Trace:
W20250113 08:59:19.021847 12717 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.010s sys 0.011s
I20250113 08:59:19.282325 14580 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.283809 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 600) took 1409 ms. Trace:
I20250113 08:59:19.102380 14736 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:19.052512 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:19.286162 11964 raft_consensus.cc:3049] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:19.286499 11964 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:19.287070 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:19.287209 11964 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:19.011466 12721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 817) took 2665 ms (client timeout 2167 ms). Trace:
I20250113 08:59:19.045922 14802 raft_consensus.cc:695] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:19.288375 11964 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:19.290179 14460 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:19.289757 14802 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.292347 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 563) took 1464 ms. Trace:
I20250113 08:59:19.292733 11887 rpcz_store.cc:276] 0113 08:59:17.828096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.828225 (+ 129us) service_pool.cc:225] Handling call
0113 08:59:17.828536 (+ 311us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:18.375063 (+546527us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:19.292124 (+917061us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b5400090454. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:19.292286 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1208645504}
W20250113 08:59:19.294711 14718 consensus_meta.cc:287] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.572s user 0.009s sys 0.027s
I20250113 08:59:18.982280 14798 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.297942 14798 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:19.046314 14787 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:19.279098 14211 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:19.282312 12033 rpcz_store.cc:276] 0113 08:59:17.754979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.755294 (+ 315us) service_pool.cc:225] Handling call
0113 08:59:17.792537 (+ 37243us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:17.967738 (+175201us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:17.967766 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:17.975638 (+ 7872us) log.cc:844] Serialized 52 byte log entry
0113 08:59:18.507301 (+531663us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:18.507387 (+ 86us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:18.507463 (+ 76us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:19.010050 (+502587us) raft_consensus.cc:1680] finished
0113 08:59:19.010080 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:19.229024 (+218944us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":673,"raft.queue_time_us":56572,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":8505,"thread_start_us":587704,"threads_started":2,"wal-append.queue_time_us":915805}
I20250113 08:59:18.990403 14800 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:19.307307 11964 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250113 08:59:18.997087 14795 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.407608 14211 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:19.226513 14421 consensus_meta.cc:287] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.764s user 0.011s sys 0.000s
I20250113 08:59:19.284286 12717 rpcz_store.cc:276] 0113 08:59:17.874619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.031248 (+156629us) service_pool.cc:225] Handling call
0113 08:59:18.222227 (+190979us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:18.299252 (+ 77025us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:18.304142 (+ 4890us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:18.393668 (+ 89526us) log.cc:844] Serialized 52 byte log entry
0113 08:59:19.283310 (+889642us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:19.283379 (+ 69us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:19.283472 (+ 93us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:19.283525 (+ 53us) raft_consensus.cc:1680] finished
0113 08:59:19.283548 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:19.283742 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":134685,"spinlock_wait_cycles":62193408,"thread_start_us":889126,"threads_started":1,"wal-append.queue_time_us":597961}
I20250113 08:59:19.376245 14539 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:19.272902 14609 consensus_meta.cc:287] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.818s user 0.007s sys 0.003s
I20250113 08:59:19.411053 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:19.234102 14789 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:19.227627 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2242) took 1263 ms. Trace:
W20250113 08:59:19.366681 12006 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.006s sys 0.014s
I20250113 08:59:19.409580 11964 raft_consensus.cc:1270] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250113 08:59:19.403602 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.026s sys 0.076s
W20250113 08:59:19.417794 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.026s sys 0.076s
I20250113 08:59:19.412113 12717 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:19.362776 11963 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
W20250113 08:59:19.413765 12006 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.007s sys 0.015s
W20250113 08:59:19.437819 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 697926us
W20250113 08:59:19.117381 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 55601.7us
I20250113 08:59:19.229715 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2211) took 1662 ms. Trace:
I20250113 08:59:19.455017 11961 rpcz_store.cc:276] 0113 08:59:17.566858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.567039 (+ 181us) service_pool.cc:225] Handling call
0113 08:59:19.229459 (+1662420us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54002a3354. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:19.229613 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2303396736}
W20250113 08:59:19.404016 12686 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.009s sys 0.005s
W20250113 08:59:19.456406 12686 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.009s sys 0.005s
W20250113 08:59:19.228122 14768 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20250113 08:59:19.413030 12042 rpcz_store.cc:276] 0113 08:59:17.964018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.964178 (+ 160us) service_pool.cc:225] Handling call
0113 08:59:17.977948 (+ 13770us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:17.978338 (+ 390us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:17.978368 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:17.979043 (+ 675us) log.cc:844] Serialized 52 byte log entry
0113 08:59:18.189291 (+210248us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:18.189362 (+ 71us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:18.189443 (+ 81us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:19.009861 (+820418us) raft_consensus.cc:1680] finished
0113 08:59:19.009884 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:19.227554 (+217670us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5126,"raft.queue_time_us":20520,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":16414,"thread_start_us":209768,"threads_started":1,"wal-append.queue_time_us":680006}
W20250113 08:59:19.143177 14291 consensus_meta.cc:287] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.752s user 0.004s sys 0.009s
W20250113 08:59:19.214885 14123 consensus_meta.cc:287] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.121s user 0.013s sys 0.001s
W20250113 08:59:19.227320 14589 consensus_meta.cc:287] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.339s user 0.006s sys 0.009s
I20250113 08:59:19.475013 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:19.475767 11963 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:19.477841 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:19.044018 11962 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:19.478892 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 64412.8us
I20250113 08:59:19.481062 11893 raft_consensus.cc:1270] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
W20250113 08:59:19.227324 14664 consensus_meta.cc:287] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.929s user 0.010s sys 0.002s
I20250113 08:59:19.411249 14609 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:19.484074 12007 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:19.375622 14749 consensus_meta.cc:287] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.623s user 0.010s sys 0.000s
I20250113 08:59:19.412874 14800 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:19.233836 14769 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:19.488284 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:19.481505 11893 pending_rounds.cc:77] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 2
I20250113 08:59:19.229437 14777 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:19.268939 14522 consensus_queue.cc:1225] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250113 08:59:19.301949 14718 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:19.044787 14723 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:19.045650 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:19.101661 14788 raft_consensus.cc:2758] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250113 08:59:19.232359 14779 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:19.171352 14792 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:19.142742 14824 consensus_queue.cc:1046] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.365s
I20250113 08:59:19.494251 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:19.611238 11884 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:19.485878 14805 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:19.547134 11893 pending_rounds.cc:99] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250113 08:59:19.508404 12007 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:19.616070 11893 raft_consensus.cc:2883] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:19.547585 11888 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:19.546298 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:19.509001 12006 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:19.553057 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:19.513271 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:19.617761 12724 raft_consensus.cc:2405] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:19.617905 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 109030us
I20250113 08:59:19.618275 11964 raft_consensus.cc:2388] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:19.618708 12006 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:19.288165 12721 rpcz_store.cc:269] 0113 08:59:16.343787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:16.343905 (+ 118us) service_pool.cc:225] Handling call
0113 08:59:17.345966 (+1002061us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:19.009014 (+1663048us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54006f2a54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:19.009179 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378558464}
I20250113 08:59:19.479780 14129 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:19.049441 14731 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:19.508769 11856 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:19.546274 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:19.458963 14787 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:19.626662 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 118326us
I20250113 08:59:19.556311 14750 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:19.486907 14589 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:19.560739 12684 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.017s sys 0.009s
I20250113 08:59:19.483893 14211 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:19.639966 12684 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.786s user 0.018s sys 0.013s
W20250113 08:59:19.640277 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 795906us
W20250113 08:59:19.591827 11859 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.007s sys 0.027s
I20250113 08:59:19.486748 14123 raft_consensus.cc:2743] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:19.011476 14778 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:19.608059 11965 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.020s sys 0.017s
W20250113 08:59:19.603313 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 92278.5us
W20250113 08:59:19.556591 12718 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.011s sys 0.014s
I20250113 08:59:19.509267 11967 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
W20250113 08:59:19.645802 11965 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s user 0.021s sys 0.017s
I20250113 08:59:19.492568 14560 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.137s
I20250113 08:59:19.611989 14580 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:19.510430 14798 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:19.619913 12006 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:19.555229 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:19.493280 14800 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:19.558748 14610 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:19.551076 14827 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:19.510543 14725 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:19.678683 12685 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:19.638197 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 129545us
W20250113 08:59:19.641543 11859 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.836s user 0.007s sys 0.027s
I20250113 08:59:19.488296 14664 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:19.478714 14527 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:19.652606 11961 raft_consensus.cc:1270] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 08:59:19.650990 14129 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:19.631394 14562 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.024s sys 0.043s
I20250113 08:59:19.636451 14805 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.587843 14318 consensus_queue.cc:1035] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20250113 08:59:19.637663 14779 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:19.667763 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:19.626217 14792 raft_consensus.cc:2743] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:19.493521 14749 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.756721 11966 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:19.507477 14823 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:19.275579 14776 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:19.651266 12718 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.853s user 0.011s sys 0.014s
I20250113 08:59:19.820919 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:19.639199 14150 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:19.702389 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:19.667883 14609 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:19.628319 14242 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.939s user 0.024s sys 0.045s
I20250113 08:59:19.681813 11968 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:19.643975 14393 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:19.697577 14562 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.024s sys 0.043s
I20250113 08:59:19.694074 14527 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:19.718603 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.71086e+06us
I20250113 08:59:19.698211 14788 consensus_queue.cc:1046] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20250113 08:59:19.701112 14610 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.650907 14522 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:19.651855 14560 consensus_queue.cc:1225] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20250113 08:59:19.681442 14621 consensus_meta.cc:287] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.075s user 0.002s sys 0.010s
W20250113 08:59:19.720333 14803 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.036s sys 0.052s
I20250113 08:59:19.720216 14800 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.720376 14821 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:19.734129 14779 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:19.681145 14798 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.702883 14795 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:19.627096 14815 consensus_queue.cc:1046] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.114s
W20250113 08:59:19.820312 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.05176e+06us
I20250113 08:59:19.818935 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:19.821856 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:19.828827 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:19.892438 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 248481us
I20250113 08:59:19.893605 12037 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:19.555995 14810 consensus_queue.cc:1046] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.460s
I20250113 08:59:19.638648 14787 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.710275 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:19.702020 14725 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:19.647053 14750 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:19.942632 12040 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.005s sys 0.005s
W20250113 08:59:20.707461 14242 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 2.026s user 0.034s sys 0.048s
I20250113 08:59:20.216365 14827 raft_consensus.cc:695] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:20.708421 14242 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.705834 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 612) took 1100 ms. Trace:
I20250113 08:59:20.709144 14827 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:20.709894 12724 rpcz_store.cc:276] 0113 08:59:19.606027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.667708 (+ 61681us) service_pool.cc:225] Handling call
0113 08:59:19.668019 (+ 311us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:19.668209 (+ 190us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:19.668233 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:19.668274 (+ 41us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:19.668308 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:20.705610 (+1037302us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:20.705770 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:20.710633 14150 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:20.712177 14609 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.712062 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:20.712066 14725 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:20.712914 12724 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
W20250113 08:59:20.384570 12008 connection.cc:232] RPC call timeout handler was delayed by -1.1631291520000104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
W20250113 08:59:20.150352 14724 consensus_meta.cc:287] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.154s user 0.007s sys 0.007s
I20250113 08:59:20.715039 14393 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.735411 11890 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20250113 08:59:20.735839 11890 pending_rounds.cc:77] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 9
I20250113 08:59:20.735540 14393 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:20.714273 14473 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.261855 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 749) took 2335 ms. Trace:
I20250113 08:59:20.737007 14473 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:20.739168 14473 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:20.737272 12718 rpcz_store.cc:276] 0113 08:59:17.927327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.350147 (+422820us) service_pool.cc:225] Handling call
0113 08:59:18.402867 (+ 52720us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:18.460556 (+ 57689us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:18.460591 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:18.461122 (+ 531us) log.cc:844] Serialized 26 byte log entry
0113 08:59:18.731532 (+270410us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:20.016171 (+1284639us) log.cc:844] Serialized 14 byte log entry
0113 08:59:20.198792 (+182621us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:20.261352 (+ 62560us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:20.261457 (+ 105us) raft_consensus.cc:1680] finished
0113 08:59:20.261488 (+ 31us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:20.261761 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8727,"prepare.queue_time_us":439665,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":7629,"raft.queue_time_us":245512,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":193,"thread_start_us":1250813,"threads_started":2,"wal-append.queue_time_us":825497}
I20250113 08:59:20.714051 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:19.751713 14767 consensus_meta.cc:287] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.166s user 0.012s sys 0.000s
W20250113 08:59:20.715981 14803 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.771s user 0.051s sys 0.067s
I20250113 08:59:20.716595 14560 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.254662 14776 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:19.717545 12720 raft_consensus.cc:1235] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250113 08:59:20.718950 14798 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:20.719211 14610 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:20.720037 11891 raft_consensus.cc:1235] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:20.720973 14800 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:20.720850 11859 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:20.027873 14729 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.721829 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2249) took 1037 ms. Trace:
W20250113 08:59:20.722970 11858 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.009s sys 0.019s
I20250113 08:59:20.723368 14750 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:20.019771 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:20.169629 11888 consensus_meta.cc:287] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.553s user 0.009s sys 0.000s
I20250113 08:59:20.635149 14749 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:20.724061 14522 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:20.016563 14805 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:20.697540 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.995s user 0.004s sys 0.026s
I20250113 08:59:20.016885 14589 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:20.198295 14460 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.022794 14723 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:20.302189 12719 consensus_meta.cc:287] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.456s user 0.007s sys 0.006s
W20250113 08:59:20.177017 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 151791us
I20250113 08:59:19.804266 14778 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:20.726895 12036 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:20.727752 14787 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:20.017314 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:20.508067 14829 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
I20250113 08:59:20.728928 11886 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:20.398438 14746 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.703s user 0.019s sys 0.010s
I20250113 08:59:20.729652 14821 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:20.730127 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 610) took 1212 ms. Trace:
W20250113 08:59:20.705811 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.08592e+06us
W20250113 08:59:19.810500 14801 consensus_meta.cc:287] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.881s user 0.004s sys 0.007s
I20250113 08:59:20.058611 14823 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:20.733758 11931 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:20.736327 11890 pending_rounds.cc:99] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20250113 08:59:20.735949 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2241) took 1107 ms. Trace:
I20250113 08:59:19.892956 14757 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:20.707522 12040 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.473s user 0.006s sys 0.021s
I20250113 08:59:20.715041 12008 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:20.740216 14724 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:20.709038 14242 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:20.732872 14527 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:20.912717 12042 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:20.806522 11857 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.005s sys 0.014s
I20250113 08:59:20.724442 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2226) took 2149 ms. Trace:
I20250113 08:59:20.914047 11888 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
W20250113 08:59:20.913717 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.230s user 0.008s sys 0.032s
I20250113 08:59:20.913321 12037 rpcz_store.cc:276] 0113 08:59:19.684284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.684869 (+ 585us) service_pool.cc:225] Handling call
0113 08:59:20.721731 (+1036862us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:20.912876 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.892s user 0.009s sys 0.028s
W20250113 08:59:20.753818 12686 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.109s user 0.015s sys 0.015s
I20250113 08:59:20.921286 12724 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
W20250113 08:59:20.725592 14808 log.cc:926] Time spent T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.703s user 0.009s sys 0.028s
W20250113 08:59:20.923025 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 202290us
I20250113 08:59:20.947825 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 573) took 1234 ms. Trace:
I20250113 08:59:20.935776 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2206) took 1040 ms. Trace:
I20250113 08:59:20.914702 11961 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:20.948544 12006 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:20.832731 14211 consensus_meta.cc:287] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.182s user 0.009s sys 0.000s
W20250113 08:59:20.549597 14539 consensus_meta.cc:287] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.112s user 0.010s sys 0.003s
W20250113 08:59:20.938791 12717 consensus_meta.cc:287] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.444s user 0.014s sys 0.003s
I20250113 08:59:20.740442 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:21.782720 11963 rpcz_store.cc:276] 0113 08:59:19.895770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.895955 (+ 185us) service_pool.cc:225] Handling call
0113 08:59:20.935687 (+1039732us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:21.783911 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2205) took 1957 ms (client timeout 1564 ms). Trace:
W20250113 08:59:21.462746 12033 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.009s sys 0.014s
I20250113 08:59:21.168118 12722 raft_consensus.cc:1235] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250113 08:59:21.783931 12042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 563) took 1966 ms (client timeout 2091 ms). Trace:
I20250113 08:59:20.714895 14731 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:20.795902 14473 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:21.947638 11857 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.645s user 0.029s sys 0.039s
W20250113 08:59:21.948309 11961 rpcz_store.cc:269] 0113 08:59:19.826233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.828471 (+ 2238us) service_pool.cc:225] Handling call
0113 08:59:21.783812 (+1955341us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:20.790544 14291 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.001s sys 0.015s
I20250113 08:59:20.778216 14769 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:20.945956 14800 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250113 08:59:21.123660 14838 log.cc:926] Time spent T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Append to log took a long time: real 0.176s user 0.002s sys 0.001s
I20250113 08:59:20.706375 14129 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:20.946316 12719 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEARNER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:21.173064 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 742) took 3338 ms. Trace:
I20250113 08:59:20.948446 14821 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:21.949579 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2232) took 2958 ms. Trace:
I20250113 08:59:21.782759 11959 rpcz_store.cc:276] 0113 08:59:18.575156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.575303 (+ 147us) service_pool.cc:225] Handling call
0113 08:59:18.585800 (+ 10497us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:18.788033 (+202233us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:18.788081 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:18.830659 (+ 42578us) log.cc:844] Serialized 52 byte log entry
0113 08:59:18.938539 (+107880us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:18.944662 (+ 6123us) log.cc:844] Serialized 14 byte log entry
0113 08:59:18.967007 (+ 22345us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:19.720108 (+753101us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54002f0b54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:19.720152 (+ 44us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:19.720198 (+ 46us) raft_consensus.cc:1680] finished
0113 08:59:19.744309 (+ 24111us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:20.724376 (+980067us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38592,"prepare.queue_time_us":5551,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":35107,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":16385,"spinlock_wait_cycles":437609856,"thread_start_us":78843,"threads_started":1,"wal-append.queue_time_us":313084,"wal-append.run_cpu_time_us":2916,"wal-append.run_wall_time_us":1191860}
I20250113 08:59:20.936288 14560 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:21.782438 11886 rpcz_store.cc:276] 0113 08:59:19.702349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.702505 (+ 156us) service_pool.cc:225] Handling call
0113 08:59:20.936590 (+1234085us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:20.937999 14787 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:20.935937 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 562) took 1294 ms. Trace:
I20250113 08:59:20.934991 14407 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:21.952315 11890 raft_consensus.cc:2883] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:21.952291 12042 rpcz_store.cc:269] 0113 08:59:19.817724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.817899 (+ 175us) service_pool.cc:225] Handling call
0113 08:59:21.783857 (+1965958us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:21.936641 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2196) took 3150 ms (client timeout 1554 ms). Trace:
W20250113 08:59:21.783222 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.04955e+06us
W20250113 08:59:21.955129 12033 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.006s user 0.019s sys 0.030s
I20250113 08:59:21.169567 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 564) took 3205 ms. Trace:
I20250113 08:59:21.253479 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2224) took 2679 ms. Trace:
I20250113 08:59:20.938925 14610 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.757421 14779 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:20.928112 14522 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:21.965158 12718 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:20.937208 14746 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.430s user 0.023s sys 0.010s
W20250113 08:59:21.957818 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.00932e+06us
I20250113 08:59:20.938223 14609 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:21.959081 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.24408e+06us
I20250113 08:59:21.949942 14793 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:20.938766 14805 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:21.965021 11888 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 567) took 2534 ms (client timeout 1734 ms). Trace:
W20250113 08:59:21.915802 12035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 556) took 3544 ms (client timeout 3145 ms). Trace:
W20250113 08:59:22.054983 11888 rpcz_store.cc:269] 0113 08:59:19.419457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.419583 (+ 126us) service_pool.cc:225] Handling call
0113 08:59:21.954012 (+2534429us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:22.055357 12035 rpcz_store.cc:269] 0113 08:59:18.371764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.372019 (+ 255us) service_pool.cc:225] Handling call
0113 08:59:21.915447 (+3543428us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54002b8c54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:21.915670 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":893031168}
I20250113 08:59:21.926474 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:22.056427 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:22.056834 12041 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:22.057018 11888 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:22.055163 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:22.002858 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 605) took 4037 ms. Trace:
W20250113 08:59:21.724608 14833 log.cc:926] Time spent T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 2.008s user 0.004s sys 0.010s
W20250113 08:59:22.004593 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2208) took 2068 ms (client timeout 1744 ms). Trace:
I20250113 08:59:22.059654 12722 rpcz_store.cc:276] 0113 08:59:17.966163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:21.167222 (+3201059us) service_pool.cc:225] Handling call
0113 08:59:21.167752 (+ 530us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:22.002396 (+834644us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.002782 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:22.003190 11964 rpcz_store.cc:276] 0113 08:59:19.628182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.670740 (+ 42558us) service_pool.cc:225] Handling call
0113 08:59:20.735280 (+1064540us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:20.735516 (+ 236us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:20.735553 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:20.735619 (+ 66us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:20.735679 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:20.735749 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:20.735877 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:22.007004 12723 rpcz_store.cc:276] 0113 08:59:17.834755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.834879 (+ 124us) service_pool.cc:225] Handling call
0113 08:59:17.835223 (+ 344us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:18.687576 (+852353us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:21.172802 (+2485226us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b5400620554. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:21.172996 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1956134144}
I20250113 08:59:20.758200 14393 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:20.960407 14798 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:22.011530 11965 rpcz_store.cc:276] 0113 08:59:18.990747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.992400 (+ 1653us) service_pool.cc:225] Handling call
0113 08:59:18.992732 (+ 332us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:18.993133 (+ 401us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:18.993165 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:18.993698 (+ 533us) log.cc:844] Serialized 26 byte log entry
0113 08:59:19.646434 (+652736us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:19.646491 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:19.646721 (+ 230us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:21.949346 (+2302625us) raft_consensus.cc:1680] finished
0113 08:59:21.949364 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:21.949501 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13659,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"thread_start_us":652306,"threads_started":1,"wal-append.queue_time_us":709504}
W20250113 08:59:21.957760 14291 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.814s user 0.016s sys 0.045s
W20250113 08:59:21.951673 12686 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.307s user 0.020s sys 0.023s
W20250113 08:59:22.073302 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 2.43306e+06us
W20250113 08:59:22.075234 12686 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.136s (SENT) [suppressed 19 similar messages]
I20250113 08:59:20.936028 14460 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:20.991600 14801 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:22.019640 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2242) took 1302 ms. Trace:
I20250113 08:59:22.080266 11959 rpcz_store.cc:276] 0113 08:59:20.716999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.018775 (+1301776us) service_pool.cc:225] Handling call
0113 08:59:22.019116 (+ 341us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:22.019297 (+ 181us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:22.019324 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:22.019365 (+ 41us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:22.019398 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.019457 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:22.019581 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:22.049640 14800 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:21.924015 14809 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 2.941s user 0.015s sys 0.015s
I20250113 08:59:21.922655 14795 raft_consensus.cc:3049] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:20.992096 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 613) took 1370 ms. Trace:
I20250113 08:59:22.023212 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:21.248522 14752 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:22.094571 11887 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:21.938308 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1939) took 4754 ms. Trace:
I20250113 08:59:21.948836 12717 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
W20250113 08:59:21.952138 12724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 754) took 1568 ms (client timeout 1609 ms). Trace:
W20250113 08:59:22.096563 12724 rpcz_store.cc:269] 0113 08:59:20.383755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:20.713773 (+330018us) service_pool.cc:225] Handling call
0113 08:59:21.952077 (+1238304us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:22.021234 11966 rpcz_store.cc:269] 0113 08:59:18.785815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.786295 (+ 480us) service_pool.cc:225] Handling call
0113 08:59:21.936335 (+3150040us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b540015de54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:21.936523 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2072761728}
I20250113 08:59:22.047514 11967 rpcz_store.cc:276] 0113 08:59:18.574499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.574631 (+ 132us) service_pool.cc:225] Handling call
0113 08:59:18.591005 (+ 16374us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:19.645561 (+1054556us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:21.253132 (+1607571us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54001ad1d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:21.253409 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2111359744}
I20250113 08:59:22.049520 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 561) took 2407 ms. Trace:
I20250113 08:59:22.093596 14795 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:22.005129 14551 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:22.049541 14610 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:22.046339 14407 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:20.948027 12721 rpcz_store.cc:276] 0113 08:59:19.521485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.651444 (+129959us) service_pool.cc:225] Handling call
0113 08:59:20.730041 (+1078597us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:22.052574 12008 connection.cc:232] RPC call timeout handler was delayed by -1.6216974949999212s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20250113 08:59:22.001801 14792 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:22.025002 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:22.054001 12007 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.262s user 0.005s sys 0.013s
W20250113 08:59:22.059334 11932 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.330s user 0.032s sys 0.050s
W20250113 08:59:22.059828 11963 rpcz_store.cc:269] 0113 08:59:19.896480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:21.964231 (+2067751us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:21.964378 (+ 147us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:22.005496 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:22.075666 14805 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:21.989552 14647 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:22.062235 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:20.935853 14729 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:21.917799 14841 log.cc:926] Time spent T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.366s user 0.004s sys 0.004s
I20250113 08:59:22.067839 12006 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:22.010506 14803 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.882s user 0.020s sys 0.042s
W20250113 08:59:22.069003 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.221s user 0.005s sys 0.009s
W20250113 08:59:22.095700 14810 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.300s user 0.018s sys 0.007s
W20250113 08:59:22.158557 14810 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.363s user 0.018s sys 0.007s
I20250113 08:59:22.158680 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 566) took 1242 ms. Trace:
I20250113 08:59:22.070115 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 614) took 2364 ms. Trace:
I20250113 08:59:20.954984 14723 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:20.937597 14767 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:22.075562 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:22.019402 14757 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:59:22.076949 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.250s user 0.009s sys 0.006s
I20250113 08:59:22.077277 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:22.019101 14527 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:22.048401 14318 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:21.964776 14731 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:21.959946 14473 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:22.050740 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 751) took 3374 ms. Trace:
I20250113 08:59:21.168409 14790 raft_consensus.cc:3049] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
W20250113 08:59:22.092187 12684 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.015s sys 0.005s
W20250113 08:59:22.092592 12687 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.268s user 0.013s sys 0.006s
I20250113 08:59:21.250448 14541 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:22.020855 12036 rpcz_store.cc:276] 0113 08:59:19.641866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.644171 (+ 2305us) service_pool.cc:225] Handling call
0113 08:59:20.928816 (+1284645us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:22.081997 14732 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20250113 08:59:22.095295 11893 rpcz_store.cc:276] 0113 08:59:17.183683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.260585 (+ 76902us) service_pool.cc:225] Handling call
0113 08:59:17.283328 (+ 22743us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:19.634953 (+2351625us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:21.937961 (+2303008us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b540037f354. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:21.938196 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5053315328}
I20250113 08:59:22.046782 14787 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:22.023573 11889 rpcz_store.cc:276] 0113 08:59:17.963984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.969190 (+ 5206us) service_pool.cc:225] Handling call
0113 08:59:17.978088 (+ 8898us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:18.142184 (+164096us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:18.142218 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:18.142259 (+ 41us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:18.142300 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:21.169214 (+3026914us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b5400060ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:21.169302 (+ 88us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:21.169491 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":376913280}
I20250113 08:59:22.011994 14769 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:22.077970 14798 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:21.251672 14344 raft_consensus.cc:3049] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:22.067188 14793 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:22.159196 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.311s user 0.005s sys 0.010s
W20250113 08:59:22.670431 12684 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.752s user 0.023s sys 0.022s
I20250113 08:59:22.342433 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 570) took 2857 ms. Trace:
W20250113 08:59:22.606710 12041 consensus_meta.cc:287] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.539s user 0.008s sys 0.002s
I20250113 08:59:22.671983 12041 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:22.148267 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 750) took 4176 ms. Trace:
I20250113 08:59:22.670800 12035 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:22.671453 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.847s user 0.013s sys 0.021s
W20250113 08:59:22.159807 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 92108.3us
W20250113 08:59:22.669281 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.8214e+06us
W20250113 08:59:22.671861 12008 leader_election.cc:336] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.592s (SENT)
I20250113 08:59:22.108148 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2227) took 3532 ms. Trace:
I20250113 08:59:22.159178 12042 rpcz_store.cc:276] 0113 08:59:20.916387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.021271 (+1104884us) service_pool.cc:225] Handling call
0113 08:59:22.158597 (+137326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:22.120018 14560 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.075066 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:22.712687 12008 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:22.097962 12033 rpcz_store.cc:276] 0113 08:59:19.641594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.642025 (+ 431us) service_pool.cc:225] Handling call
0113 08:59:19.667199 (+ 25174us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:20.263151 (+595952us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:20.263197 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:20.706641 (+443444us) log.cc:844] Serialized 52 byte log entry
0113 08:59:20.881834 (+175193us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:22.048743 (+1166909us) log.cc:844] Serialized 14 byte log entry
0113 08:59:22.049016 (+ 273us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.049216 (+ 200us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:22.049277 (+ 61us) raft_consensus.cc:1680] finished
0113 08:59:22.049290 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:22.049439 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":311992,"prepare.queue_time_us":1186035,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1853,"raft.queue_time_us":593276,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":16317,"thread_start_us":1274313,"threads_started":2,"wal-append.queue_time_us":423006}
I20250113 08:59:22.675580 11934 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:22.673302 11884 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:23.076149 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:22.674077 12725 rpcz_store.cc:276] 0113 08:59:17.972206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.615530 (+643324us) service_pool.cc:225] Handling call
0113 08:59:18.615875 (+ 345us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:18.616225 (+ 350us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:18.616255 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:18.622934 (+ 6679us) log.cc:844] Serialized 26 byte log entry
0113 08:59:18.768949 (+146015us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:18.939323 (+170374us) log.cc:844] Serialized 14 byte log entry
0113 08:59:18.941977 (+ 2654us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:18.942080 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:21.923085 (+2981005us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:22.125944 (+202859us) raft_consensus.cc:1680] finished
0113 08:59:22.147944 (+ 22000us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:22.148147 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":599,"prepare.queue_time_us":483513,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":199181,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":14293,"spinlock_wait_cycles":55012096,"thread_start_us":222698,"threads_started":2,"wal-append.queue_time_us":279611}
I20250113 08:59:22.714231 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:22.091609 14129 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:22.675351 12006 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:23.077517 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 576) took 1061 ms. Trace:
I20250113 08:59:22.677623 11968 rpcz_store.cc:276] 0113 08:59:18.575468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.577686 (+ 2218us) service_pool.cc:225] Handling call
0113 08:59:18.601141 (+ 23455us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:20.725214 (+2124073us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.107817 (+1382603us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001a0ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:22.108069 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2478601472}
I20250113 08:59:22.674010 12723 rpcz_store.cc:276] 0113 08:59:19.705999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.069009 (+2363010us) service_pool.cc:225] Handling call
0113 08:59:22.069416 (+ 407us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:22.069694 (+ 278us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:22.069731 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:22.069785 (+ 54us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:22.069842 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.069911 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:22.070041 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:22.674357 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 568) took 1756 ms. Trace:
I20250113 08:59:23.078589 11893 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:22.671453 11890 rpcz_store.cc:276] 0113 08:59:19.484681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.484805 (+ 124us) service_pool.cc:225] Handling call
0113 08:59:19.508200 (+ 23395us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:22.046208 (+2538008us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.342105 (+295897us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000a4454. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:22.342352 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2821607808}
I20250113 08:59:23.076061 12036 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
W20250113 08:59:22.439952 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 380811us
I20250113 08:59:23.093753 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2257) took 1022 ms. Trace:
I20250113 08:59:22.674353 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:59:22.781603 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:23.094179 11884 rpcz_store.cc:276] 0113 08:59:22.016011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.024701 (+ 8690us) service_pool.cc:225] Handling call
0113 08:59:23.077451 (+1052750us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:22.735949 12687 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.911s user 0.022s sys 0.024s
I20250113 08:59:22.106817 12037 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
W20250113 08:59:22.673874 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2248) took 2990 ms (client timeout 1844 ms). Trace:
W20250113 08:59:22.731585 12685 connection.cc:232] RPC call timeout handler was delayed by -1.7509248079999225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1734s for a response
I20250113 08:59:22.759317 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:22.672057 12007 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.880s user 0.010s sys 0.028s
W20250113 08:59:23.077478 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 401958us
I20250113 08:59:22.467509 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:22.751272 12039 raft_consensus.cc:1270] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250113 08:59:22.673601 11888 consensus_meta.cc:287] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.602s user 0.013s sys 0.000s
I20250113 08:59:22.338310 11962 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:23.096057 11932 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.367s user 0.042s sys 0.066s
W20250113 08:59:22.686156 12684 connection.cc:232] RPC call timeout handler was delayed by -1.5206878859999051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3145s for a response
I20250113 08:59:23.116905 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2244) took 4810 ms. Trace:
W20250113 08:59:23.118403 12684 connection.cc:232] RPC call timeout handler was delayed by -1.1486447009999665s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
W20250113 08:59:22.444072 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 422491us
W20250113 08:59:23.119056 12684 connection.cc:232] RPC call timeout handler was delayed by -1.1027447009998923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
I20250113 08:59:23.119745 11931 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:22.269958 14766 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:23.128374 14783 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:22.642369 14150 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:23.128746 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 456900us
W20250113 08:59:22.198068 14827 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.440s user 0.020s sys 0.023s
W20250113 08:59:22.297456 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 222227us
I20250113 08:59:23.045208 14805 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:22.119113 14460 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:22.749300 11960 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:23.119918 12684 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:23.120992 12037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 567) took 2203 ms (client timeout 1577 ms). Trace:
I20250113 08:59:23.123482 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:23.131413 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1827) took 1072 ms. Trace:
I20250113 08:59:23.167189 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:23.127113 12041 rpcz_store.cc:269] 0113 08:59:19.683983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.684784 (+ 801us) service_pool.cc:225] Handling call
0113 08:59:21.926265 (+2241481us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:22.673808 (+747543us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10655488}
I20250113 08:59:22.159741 14860 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:23.125573 12042 rpcz_store.cc:276] 0113 08:59:22.072078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.075857 (+1003779us) service_pool.cc:225] Handling call
0113 08:59:23.093685 (+ 17828us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.098007 12035 rpcz_store.cc:276] 0113 08:59:20.917866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.056486 (+1138620us) service_pool.cc:225] Handling call
0113 08:59:22.674301 (+617815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.126827 11885 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:23.127894 14788 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:23.100478 14129 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:22.104730 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 753) took 3049 ms (client timeout 1565 ms). Trace:
I20250113 08:59:22.891395 14609 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:23.042886 14872 log.cc:926] Time spent T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.214s user 0.000s sys 0.000s
I20250113 08:59:22.093421 14800 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:22.620808 12718 raft_consensus.cc:2405] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:23.117101 11888 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:23.166407 14150 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:23.132115 14647 raft_consensus.cc:2743] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:23.167184 12037 rpcz_store.cc:269] 0113 08:59:20.917146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:21.994969 (+1077823us) service_pool.cc:225] Handling call
0113 08:59:23.120892 (+1125923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.104102 12723 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:23.169696 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:23.129959 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 662568us
I20250113 08:59:22.684718 14537 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:22.674695 12719 rpcz_store.cc:276] 0113 08:59:18.678146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.741090 (+ 62944us) service_pool.cc:225] Handling call
0113 08:59:18.741692 (+ 602us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:22.006734 (+3265042us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.046576 (+ 39842us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:22.685999 14551 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.194200 14537 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.194653 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:23.196275 14537 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:23.196408 12752 maintenance_manager.cc:444] Time spent finding best maintenance operation: real 10.827s user 0.077s sys 0.049s
I20250113 08:59:23.182334 14759 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:22.750002 14792 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:22.161166 14733 raft_consensus.cc:3049] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:23.172866 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1819) took 1115 ms. Trace:
I20250113 08:59:23.198068 14729 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:22.674541 14803 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.550s user 0.029s sys 0.053s
W20250113 08:59:22.440760 14736 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 2.710s user 0.017s sys 0.020s
I20250113 08:59:23.198099 11885 rpcz_store.cc:276] 0113 08:59:22.057141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.734355 (+677214us) service_pool.cc:225] Handling call
0113 08:59:23.172807 (+438452us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.128191 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:22.671641 14790 raft_consensus.cc:738] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:23.191219 14150 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:22.716814 14461 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:22.714212 14757 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.046973 14473 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:23.202122 14736 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 3.481s user 0.025s sys 0.029s
W20250113 08:59:23.086021 14770 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.000s sys 0.008s
I20250113 08:59:22.152482 14801 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.155570 11858 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:23.092962 14865 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:23.092969 14560 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:23.215685 14647 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:22.692440 14344 raft_consensus.cc:738] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:23.180958 11932 connection.cc:232] RPC call timeout handler was delayed by -2.2551008520000551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
W20250113 08:59:23.205793 12718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 756) took 2820 ms (client timeout 2213 ms). Trace:
I20250113 08:59:23.092056 14488 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:22.749321 14828 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:22.121523 14858 consensus_queue.cc:1035] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 1.059s
W20250113 08:59:23.237349 12718 rpcz_store.cc:269] 0113 08:59:20.384530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:20.740083 (+355553us) service_pool.cc:225] Handling call
0113 08:59:23.202858 (+2462775us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:23.182065 11934 connection.cc:232] RPC call timeout handler was delayed by -1.9082726719999528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
I20250113 08:59:23.128950 14783 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.236231 14551 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.239665 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:22.815245 14474 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:22.328092 14242 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:22.690287 14823 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.128739 14766 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:23.241658 14610 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.241144 14474 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:23.186162 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 572) took 3678 ms. Trace:
I20250113 08:59:23.212750 14790 consensus_queue.cc:260] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.130609 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:23.169529 14827 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 2.456s user 0.040s sys 0.065s
I20250113 08:59:23.113924 14767 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.187885 14601 consensus_queue.cc:1046] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:23.243971 14601 consensus_queue.cc:1225] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer 5f9c33777f6845c2848a2a6102a1c2ff log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:23.168670 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2253) took 1111 ms. Trace:
I20250113 08:59:22.093678 12720 rpcz_store.cc:276] 0113 08:59:19.622076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.678534 (+ 56458us) service_pool.cc:225] Handling call
0113 08:59:19.717286 (+ 38752us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:20.922487 (+1205201us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:20.961844 (+ 39357us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:22.592293 14779 consensus_meta.cc:287] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.529s user 0.006s sys 0.002s
I20250113 08:59:23.128969 14778 raft_consensus.cc:3049] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:23.170831 14867 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:23.192310 12722 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:23.255735 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:23.256578 12720 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:23.254359 14779 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.257151 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 615) took 3285 ms. Trace:
I20250113 08:59:23.246869 14767 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:23.257652 12722 rpcz_store.cc:276] 0113 08:59:19.971945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.067785 (+2095840us) service_pool.cc:225] Handling call
0113 08:59:23.192133 (+1124348us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.255602 (+ 63469us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.256939 (+ 1337us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400501250. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:23.257089 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2542768384}
I20250113 08:59:23.258620 12038 raft_consensus.cc:2388] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:23.192317 12041 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:23.196146 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:23.168311 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:23.260422 11884 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:59:23.260577 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2202) took 3974 ms (client timeout 2071 ms). Trace:
I20250113 08:59:23.192381 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 758) took 2807 ms. Trace:
I20250113 08:59:23.167788 11892 rpcz_store.cc:276] 0113 08:59:22.059142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.758939 (+699797us) service_pool.cc:225] Handling call
0113 08:59:23.131353 (+372414us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.197117 14129 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.197579 14759 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:23.263029 14759 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:23.263480 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:23.264276 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1832) took 1190 ms. Trace:
I20250113 08:59:23.264500 14407 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.264570 11884 rpcz_store.cc:276] 0113 08:59:22.074169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.263218 (+1189049us) service_pool.cc:225] Handling call
0113 08:59:23.264223 (+ 1005us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.180207 14788 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.266319 11884 raft_consensus.cc:1235] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:23.266772 14788 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.266964 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1814) took 1211 ms. Trace:
I20250113 08:59:23.267156 14793 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.267398 11884 rpcz_store.cc:276] 0113 08:59:22.055891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.265467 (+1209576us) service_pool.cc:225] Handling call
0113 08:59:23.266083 (+ 616us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:23.266707 (+ 624us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.266912 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.268483 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 617) took 3126 ms. Trace:
I20250113 08:59:23.268843 12724 rpcz_store.cc:276] 0113 08:59:20.141961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.104906 (+1962945us) service_pool.cc:225] Handling call
0113 08:59:22.117776 (+ 12870us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:22.754473 (+636697us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:22.754522 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:22.754578 (+ 56us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:22.754624 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.268265 (+513641us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54004105d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:23.268316 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.268429 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1463671680}
I20250113 08:59:23.269301 11884 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20250113 08:59:23.271030 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1818) took 1214 ms. Trace:
I20250113 08:59:23.271281 14541 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:23.271518 11884 rpcz_store.cc:276] 0113 08:59:22.056872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.268665 (+1211793us) service_pool.cc:225] Handling call
0113 08:59:23.269002 (+ 337us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:23.270770 (+ 1768us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.270983 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.197813 14792 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:23.274189 12033 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
W20250113 08:59:22.690450 14838 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.007s sys 0.006s
W20250113 08:59:23.274973 14838 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.122s user 0.011s sys 0.013s
I20250113 08:59:23.172190 14769 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:23.182785 14878 log.cc:926] Time spent T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
W20250113 08:59:22.736182 14522 consensus_meta.cc:287] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.674s user 0.011s sys 0.000s
I20250113 08:59:22.685231 14527 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:23.174961 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 55101us
I20250113 08:59:23.181797 14609 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:22.714718 14318 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:22.330799 14847 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.858s
I20250113 08:59:23.178808 14787 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.201874 14729 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:23.318310 14729 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:23.318759 14729 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:23.045743 14731 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.322392 12720 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:23.254850 14778 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:23.190383 14860 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:23.046054 14723 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.197782 14733 raft_consensus.cc:738] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:22.527356 14821 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:23.342063 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 564) took 3524 ms (client timeout 2810 ms). Trace:
I20250113 08:59:23.342248 14821 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:23.342602 12038 rpcz_store.cc:269] 0113 08:59:19.818101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.818582 (+ 481us) service_pool.cc:225] Handling call
0113 08:59:23.341803 (+3523221us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b54000cd0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:23.341980 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5822406528}
I20250113 08:59:23.343268 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2250) took 3658 ms. Trace:
W20250113 08:59:23.177975 12717 rpcz_store.cc:269] 0113 08:59:19.049512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.410543 (+361031us) service_pool.cc:225] Handling call
0113 08:59:22.098645 (+2688102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.343708 12039 rpcz_store.cc:276] 0113 08:59:19.684643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.684766 (+ 123us) service_pool.cc:225] Handling call
0113 08:59:22.611761 (+2926995us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:59:22.612062 (+ 301us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.116919 (+504857us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.343041 (+226122us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54002afb54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:23.343209 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":330846336}
I20250113 08:59:23.207537 14150 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:23.346582 11967 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250113 08:59:23.118503 12040 rpcz_store.cc:276] 0113 08:59:18.269899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.270109 (+ 210us) service_pool.cc:225] Handling call
0113 08:59:18.307976 (+ 37867us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:18.371008 (+ 63032us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:18.371038 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:19.233510 (+862472us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b2c0011cef8. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc334fea741 00007fc334ff9d72 00007fc336645141 00007fc33660958a 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782
0113 08:59:19.234187 (+ 677us) log.cc:844] Serialized 52 byte log entry
0113 08:59:21.954536 (+2720349us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:22.306275 (+351739us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b180035ed20. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d10d8 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:59:22.474852 (+168577us) log.cc:844] Serialized 14 byte log entry
0113 08:59:22.815107 (+340255us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.822034 (+ 6927us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:22.822115 (+ 81us) raft_consensus.cc:1680] finished
0113 08:59:22.822133 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.079906 (+257773us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49492,"prepare.queue_time_us":180108,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":12711,"raft.queue_time_us":80050,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":18449,"spinlock_wait_cycles":59026048,"thread_start_us":3523492,"threads_started":3,"wal-append.queue_time_us":1645751,"wal-append.run_cpu_time_us":4113,"wal-append.run_wall_time_us":1222832}
I20250113 08:59:23.349892 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:23.350566 12040 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
W20250113 08:59:23.350782 14874 log.cc:926] Time spent T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.163s user 0.003s sys 0.000s
I20250113 08:59:23.351589 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:23.352444 12040 raft_consensus.cc:2405] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:23.215516 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:23.353922 11885 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:23.224056 14770 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.931s user 0.002s sys 0.008s
I20250113 08:59:23.226881 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:23.233513 14560 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:22.319487 14809 log.cc:926] Time spent T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250113 08:59:22.814088 14795 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.129452 14727 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:23.240533 14783 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.324690 14847 consensus_queue.cc:1225] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:23.185653 11893 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
W20250113 08:59:23.384449 11893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1831) took 1324 ms (client timeout 1652 ms). Trace:
W20250113 08:59:23.385012 11893 rpcz_store.cc:269] 0113 08:59:22.060224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.713864 (+653640us) service_pool.cc:225] Handling call
0113 08:59:23.384378 (+670514us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.241887 14766 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:23.186507 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1815) took 1130 ms. Trace:
I20250113 08:59:22.123706 14664 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.242655 14790 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:23.242336 11891 rpcz_store.cc:276] 0113 08:59:19.507315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.509759 (+ 2444us) service_pool.cc:225] Handling call
0113 08:59:19.558742 (+ 48983us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:20.914086 (+1355344us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.185892 (+2271806us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540034cbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:23.186098 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2670201728}
I20250113 08:59:23.386296 11888 rpcz_store.cc:276] 0113 08:59:22.056143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.056210 (+ 67us) service_pool.cc:225] Handling call
0113 08:59:23.186444 (+1130234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.243010 14798 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.232479 14865 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:23.236665 14791 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 08:59:23.245324 14827 raft_consensus.cc:3049] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:23.245790 14828 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.247437 14800 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.255580 14867 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.166812 14421 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:23.008679 14807 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:23.261153 11962 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:23.402000 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1821) took 1333 ms. Trace:
W20250113 08:59:23.260969 11960 rpcz_store.cc:269] 0113 08:59:19.286282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.286819 (+ 537us) service_pool.cc:225] Handling call
0113 08:59:23.260437 (+3973618us) spinlock_profiling.cc:243] Waited 3.46 s on lock 0x7b5400380254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:23.260518 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7961175424}
I20250113 08:59:23.261097 12723 rpcz_store.cc:276] 0113 08:59:20.385148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.103224 (+2718076us) service_pool.cc:225] Handling call
0113 08:59:23.103648 (+ 424us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.192127 (+ 88479us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.192320 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.281653 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:23.287214 12041 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:23.213184 14461 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.258226 14757 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.305379 12684 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:23.413007 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 107709us
W20250113 08:59:23.307396 14718 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 3.573s user 0.016s sys 0.039s
W20250113 08:59:23.308104 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 108659us
W20250113 08:59:23.316473 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 196821us
I20250113 08:59:23.317392 14609 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.189941 14805 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:23.302484 14522 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:23.334853 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:23.247797 12036 rpcz_store.cc:276] 0113 08:59:22.047131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.756266 (+709135us) service_pool.cc:225] Handling call
0113 08:59:23.158764 (+402498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.336290 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:23.337571 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:23.435992 11965 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:23.436169 11968 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:23.338239 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:23.338155 14647 consensus_queue.cc:1046] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:23.343922 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:23.422654 14718 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 3.693s user 0.016s sys 0.039s
I20250113 08:59:23.231070 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 555) took 4923 ms. Trace:
W20250113 08:59:23.232209 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 76745.3us
I20250113 08:59:23.345099 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 566) took 3926 ms. Trace:
I20250113 08:59:23.441592 11887 rpcz_store.cc:276] 0113 08:59:19.419059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:19.419179 (+ 120us) service_pool.cc:225] Handling call
0113 08:59:19.492927 (+ 73748us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:22.094993 (+2602066us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.344818 (+1249825us) spinlock_profiling.cc:243] Waited 2.6 s on lock 0x7b540037ee54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:23.345029 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5982951680}
I20250113 08:59:23.315415 14473 consensus_queue.cc:1046] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:23.272964 14759 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.345085 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:23.367381 12033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2255) took 1309 ms (client timeout 1744 ms). Trace:
W20250113 08:59:23.493803 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 63174.3us
W20250113 08:59:23.494159 12033 rpcz_store.cc:269] 0113 08:59:22.058181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.079862 (+1021681us) service_pool.cc:225] Handling call
0113 08:59:23.367165 (+287303us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54002b9154. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:23.367304 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":244294400}
I20250113 08:59:23.235253 14715 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:23.365372 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2207) took 3469 ms. Trace:
I20250113 08:59:23.285348 14767 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.253793 14344 consensus_queue.cc:260] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.405565 11892 raft_consensus.cc:2405] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:23.406595 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2256) took 1341 ms. Trace:
I20250113 08:59:23.514727 12041 rpcz_store.cc:276] 0113 08:59:22.064803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.169445 (+1104642us) service_pool.cc:225] Handling call
0113 08:59:23.406537 (+237092us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.407120 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:23.408138 11891 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250113 08:59:23.404276 11893 rpcz_store.cc:276] 0113 08:59:22.057667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.390237 (+1332570us) service_pool.cc:225] Handling call
0113 08:59:23.390626 (+ 389us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.390849 (+ 223us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:23.390886 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:23.390943 (+ 57us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:23.390989 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.391059 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.391214 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.410033 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2199) took 4456 ms. Trace:
I20250113 08:59:23.516448 11962 rpcz_store.cc:276] 0113 08:59:18.948572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.948700 (+ 128us) service_pool.cc:225] Handling call
0113 08:59:18.982677 (+ 33977us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:22.338131 (+3355454us) spinlock_profiling.cc:243] Waited 2.85 s on lock 0x7b580082f7b0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc3365851c3 00007fc3365e8aba 00007fc3366003af 00007fc336601b4e 00007fc33660bc89 00007fc33660c556 00007fc336608edf 00007fc33660899e 00007fc33b5672d9 00007fc334c33694
0113 08:59:23.404182 (+1066051us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.404455 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6548560256}
I20250113 08:59:23.517750 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:23.417201 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 287238us
I20250113 08:59:23.422291 11856 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:23.520143 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 97958.2us
I20250113 08:59:23.413825 12684 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:23.390134 14865 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:23.424141 12686 connection.cc:232] RPC call timeout handler was delayed by -2.1012492469999415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.292s for a response
I20250113 08:59:23.394901 14827 raft_consensus.cc:738] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:23.409811 14421 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.445297 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:23.385322 14731 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20250113 08:59:23.540587 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1820) took 1483 ms. Trace:
I20250113 08:59:23.515470 14880 raft_consensus.cc:3049] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Stepping down as leader of term 3
W20250113 08:59:23.420944 14717 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250113 08:59:23.437297 12038 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:23.353536 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:23.439200 12007 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:23.438988 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 203253us
I20250113 08:59:23.441097 12034 rpcz_store.cc:276] 0113 08:59:18.308201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:18.308337 (+ 136us) service_pool.cc:225] Handling call
0113 08:59:18.349899 (+ 41562us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:18.873829 (+523930us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.230737 (+4356908us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b54001f13d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:23.230993 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1203307776}
I20250113 08:59:23.451192 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:23.601954 11965 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:23.601995 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:23.461310 11934 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:23.424248 12721 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:23.421322 14770 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:23.382187 14727 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:23.502179 12685 connection.cc:232] RPC call timeout handler was delayed by -1.043455338000058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
I20250113 08:59:23.375092 14860 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.424476 14798 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:23.495654 11961 rpcz_store.cc:276] 0113 08:59:19.896179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:21.951402 (+2055223us) service_pool.cc:225] Handling call
0113 08:59:21.951760 (+ 358us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:21.952104 (+ 344us) raft_consensus.cc:1529] Early marking committed up to index 4
0113 08:59:22.443885 (+491781us) log.cc:844] Serialized 14 byte log entry
0113 08:59:22.869807 (+425922us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:22.945294 (+ 75487us) log.cc:844] Serialized 26 byte log entry
0113 08:59:22.945485 (+ 191us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:22.945571 (+ 86us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:22.945681 (+ 110us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:23.350048 (+404367us) spinlock_profiling.cc:243] Waited 873 us on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:23.354504 (+ 4456us) raft_consensus.cc:1680] finished
0113 08:59:23.365087 (+ 10583us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.365288 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2152,"prepare.queue_time_us":811078,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":374796,"raft.run_cpu_time_us":431,"raft.run_wall_time_us":126927,"spinlock_wait_cycles":2007808,"thread_start_us":677916,"threads_started":2,"wal-append.queue_time_us":715239}
I20250113 08:59:23.521487 11887 raft_consensus.cc:1235] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:23.523931 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1822) took 1466 ms. Trace:
I20250113 08:59:23.524430 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1816) took 1468 ms. Trace:
W20250113 08:59:23.528563 12686 connection.cc:232] RPC call timeout handler was delayed by -1.7529767039999342s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2096s for a response
I20250113 08:59:23.525377 14561 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:23.391515 14791 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:23.530342 11856 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:23.531790 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:23.497869 14757 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:23.537581 11893 raft_consensus.cc:1235] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:23.426579 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2251) took 1407 ms. Trace:
I20250113 08:59:23.430601 14664 consensus_queue.cc:1046] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
W20250113 08:59:23.530242 14895 log.cc:926] Time spent T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.073s user 0.001s sys 0.003s
I20250113 08:59:23.541115 11884 rpcz_store.cc:276] 0113 08:59:22.057406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.272640 (+1215234us) service_pool.cc:225] Handling call
0113 08:59:23.272948 (+ 308us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.273154 (+ 206us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:23.409863 (+136709us) log.cc:844] Serialized 14 byte log entry
0113 08:59:23.540258 (+130395us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:23.540307 (+ 49us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:23.540358 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.540422 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.540530 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":138312,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"thread_start_us":266378,"threads_started":2}
W20250113 08:59:23.419483 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 744177us
I20250113 08:59:23.437312 14805 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:23.556221 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 142491us
I20250113 08:59:23.595171 12033 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250113 08:59:23.412675 14807 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.526142 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:23.537137 14792 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:23.411370 14778 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.534363 14723 consensus_queue.cc:1035] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.104s
I20250113 08:59:23.506048 14889 consensus_queue.cc:1046] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.214s
I20250113 08:59:23.875104 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1826) took 1816 ms. Trace:
I20250113 08:59:23.675616 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2254) took 1622 ms. Trace:
I20250113 08:59:23.962400 11893 rpcz_store.cc:276] 0113 08:59:22.058883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.527987 (+1469104us) service_pool.cc:225] Handling call
0113 08:59:23.537409 (+ 9422us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.874835 (+337426us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.875043 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.655347 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:23.873700 12685 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:23.712672 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:23.603070 11963 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
W20250113 08:59:23.657917 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 218776us
I20250113 08:59:23.965889 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1825) took 1907 ms. Trace:
I20250113 08:59:23.966300 11887 rpcz_store.cc:276] 0113 08:59:22.058648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.442727 (+1384079us) service_pool.cc:225] Handling call
0113 08:59:23.521314 (+ 78587us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.965639 (+444325us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.965836 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:23.871557 12686 connection.cc:232] RPC call timeout handler was delayed by -1.5540492469999663s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3528s for a response
I20250113 08:59:23.967279 12034 raft_consensus.cc:2388] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 3: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:23.967202 12686 connection.cc:232] RPC call timeout handler was delayed by -1.2732777359999545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3327s for a response
W20250113 08:59:23.967694 12686 connection.cc:232] RPC call timeout handler was delayed by -1.0075021009998864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3145s for a response
W20250113 08:59:23.597851 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 52399.5us
I20250113 08:59:23.711066 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:23.968394 12008 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:23.969978 11892 rpcz_store.cc:276] 0113 08:59:22.056372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.281311 (+1224939us) service_pool.cc:225] Handling call
0113 08:59:23.524366 (+243055us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.875871 11891 rpcz_store.cc:276] 0113 08:59:22.057948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.392781 (+1334833us) service_pool.cc:225] Handling call
0113 08:59:23.407897 (+ 15116us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:23.523654 (+115757us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.523871 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.872664 12038 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:23.749013 11885 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 8.
I20250113 08:59:23.544664 14344 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:23.975836 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:23.977752 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1828) took 1918 ms. Trace:
I20250113 08:59:23.978440 14779 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:23.603812 11932 leader_election.cc:336] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.202s (SENT)
I20250113 08:59:23.962456 12040 rpcz_store.cc:276] 0113 08:59:22.053645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.352974 (+1299329us) service_pool.cc:225] Handling call
0113 08:59:23.355559 (+ 2585us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.355789 (+ 230us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:23.356100 (+ 311us) log.cc:844] Serialized 14 byte log entry
0113 08:59:23.675104 (+319004us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:23.675181 (+ 77us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:23.675258 (+ 77us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.675359 (+ 101us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.675532 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92972,"prepare.queue_time_us":2585,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":53,"raft.queue_time_us":93913,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":225525,"threads_started":1,"wal-append.queue_time_us":367680}
I20250113 08:59:23.980168 11884 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:23.971254 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2259) took 1893 ms. Trace:
I20250113 08:59:23.543004 14731 consensus_queue.cc:1225] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20250113 08:59:23.974680 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 444400us
I20250113 08:59:23.872033 12035 rpcz_store.cc:276] 0113 08:59:22.019338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.172368 (+1153030us) service_pool.cc:225] Handling call
0113 08:59:23.184467 (+ 12099us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.184640 (+ 173us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:23.186936 (+ 2296us) log.cc:844] Serialized 14 byte log entry
0113 08:59:23.426129 (+239193us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:23.426201 (+ 72us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:23.426276 (+ 75us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.426361 (+ 85us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.426506 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89533,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":8534,"raft.queue_time_us":757,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":9313,"spinlock_wait_cycles":62848,"thread_start_us":237498,"threads_started":1,"wal-append.queue_time_us":243122}
W20250113 08:59:23.785606 14892 log.cc:926] Time spent T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
W20250113 08:59:23.602598 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 97505us
W20250113 08:59:23.964541 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 523704us
I20250113 08:59:23.983358 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1817) took 1926 ms. Trace:
W20250113 08:59:23.871141 14894 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20250113 08:59:23.985406 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:23.986567 12039 raft_consensus.cc:2388] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:23.919780 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 618) took 3505 ms. Trace:
W20250113 08:59:23.718369 11993 heartbeater.cc:643] Failed to heartbeat to 127.10.115.190:41823 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.10.115.190:41823 timed out after 15.000s (SENT)
I20250113 08:59:24.088065 11891 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:24.088840 11856 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:24.090107 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:24.091012 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 122702us
I20250113 08:59:24.091996 14421 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:24.092300 14731 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:24.093891 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:23.588831 14527 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:24.094965 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:24.095443 11932 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:24.096707 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:24.098445 14807 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:24.100749 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 129164us
I20250113 08:59:24.102658 12038 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:23.991981 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1829) took 1932 ms. Trace:
W20250113 08:59:23.719523 14885 log.cc:926] Time spent T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.176s user 0.000s sys 0.000s
I20250113 08:59:23.543375 14880 raft_consensus.cc:738] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:23.996469 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:23.991075 11893 rpcz_store.cc:276] 0113 08:59:22.059382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.976421 (+1917039us) service_pool.cc:225] Handling call
0113 08:59:23.976759 (+ 338us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.976985 (+ 226us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:23.977018 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:23.977066 (+ 48us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:23.977113 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.977169 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.977685 (+ 516us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:23.722285 14893 log.cc:926] Time spent T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.282s user 0.000s sys 0.000s
I20250113 08:59:24.606561 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1687) took 1064 ms. Trace:
I20250113 08:59:23.602444 12717 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:23.601127 14845 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:23.770025 14896 log.cc:926] Time spent T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.340s user 0.001s sys 0.000s
W20250113 08:59:23.828071 14541 consensus_meta.cc:287] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.556s user 0.006s sys 0.005s
W20250113 08:59:24.244697 12041 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.006s sys 0.064s
W20250113 08:59:24.317744 11967 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.006s sys 0.021s
W20250113 08:59:24.758020 12041 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.218s user 0.006s sys 0.072s
W20250113 08:59:24.758191 11967 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.221s user 0.019s sys 0.021s
I20250113 08:59:23.996181 11962 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:23.722983 14461 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:23.844377 14828 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:24.606897 11884 rpcz_store.cc:276] 0113 08:59:22.059668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.979690 (+1920022us) service_pool.cc:225] Handling call
0113 08:59:23.980001 (+ 311us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.991652 (+ 11651us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.991911 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.787753 12036 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:24.628456 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1690) took 1084 ms. Trace:
I20250113 08:59:24.763262 12039 rpcz_store.cc:276] 0113 08:59:23.543373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.598725 (+ 55352us) service_pool.cc:225] Handling call
0113 08:59:24.628088 (+1029363us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.719928 14560 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:24.601585 12722 rpcz_store.cc:276] 0113 08:59:20.414293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.258952 (+2844659us) service_pool.cc:225] Handling call
0113 08:59:23.260819 (+ 1867us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.589933 (+329114us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:23.589965 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:23.590007 (+ 42us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:23.590040 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.631527 (+ 41487us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540077c254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:23.631593 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:23.919666 (+288073us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756396800}
I20250113 08:59:23.718930 14150 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:24.771586 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:24.772308 14150 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:24.773036 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:23.990864 12724 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
W20250113 08:59:24.774554 12721 consensus_meta.cc:287] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.794s user 0.006s sys 0.004s
I20250113 08:59:24.774675 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:24.775314 12721 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250113 08:59:24.776113 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:23.992148 12033 rpcz_store.cc:276] 0113 08:59:22.078360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.516980 (+1438620us) service_pool.cc:225] Handling call
0113 08:59:23.594397 (+ 77417us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:59:23.594685 (+ 288us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.971071 (+376386us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:23.971213 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6069632}
I20250113 08:59:23.655536 14473 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 1.096s
I20250113 08:59:24.777896 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:24.608577 11965 raft_consensus.cc:2405] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250113 08:59:23.995780 14664 consensus_queue.cc:1225] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 5f9c33777f6845c2848a2a6102a1c2ff log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250113 08:59:24.709935 12038 rpcz_store.cc:276] 0113 08:59:23.542560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.600702 (+ 58142us) service_pool.cc:225] Handling call
0113 08:59:24.606364 (+1005662us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.983661 14798 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:24.792147 11962 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:24.215128 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:24.830101 11963 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:24.170207 12042 raft_consensus.cc:1270] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20250113 08:59:23.655933 12718 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 7: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:24.089469 11887 raft_consensus.cc:1270] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:24.722720 11885 rpcz_store.cc:276] 0113 08:59:22.056632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.205800 (+1149168us) service_pool.cc:225] Handling call
0113 08:59:23.983295 (+777495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:24.724468 11968 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:24.832675 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 579) took 2098 ms. Trace:
I20250113 08:59:23.984901 14791 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:24.838254 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:23.718822 14881 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:23.959990 14831 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:23.769101 14474 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:24.611838 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1830) took 2551 ms. Trace:
I20250113 08:59:24.714339 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1694) took 1082 ms. Trace:
I20250113 08:59:24.840476 12034 rpcz_store.cc:276] 0113 08:59:23.632466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.969101 (+336635us) service_pool.cc:225] Handling call
0113 08:59:24.714256 (+745155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:24.763898 11888 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:24.763979 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:23.714071 14609 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.979274 12720 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:24.099766 14778 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:23.988211 14887 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:24.737625 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:24.606683 14889 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.001s sys 0.015s
I20250113 08:59:23.716655 14729 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.988391 14724 raft_consensus.cc:2758] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20250113 08:59:24.726955 14561 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:24.415015 14767 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:23.995661 14757 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:24.758042 14723 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:24.757473 14845 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:24.770119 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:25.085757 12039 raft_consensus.cc:2388] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:25.086246 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1696) took 1375 ms. Trace:
W20250113 08:59:24.825784 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 104515us
I20250113 08:59:24.793833 14822 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:24.826498 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 569) took 2140 ms. Trace:
I20250113 08:59:24.827347 11889 raft_consensus.cc:2363] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:24.779393 12724 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:24.614033 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 578) took 1883 ms. Trace:
I20250113 08:59:24.804769 12007 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:24.435387 14867 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:24.833524 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2215) took 1294 ms. Trace:
I20250113 08:59:24.833709 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:24.833046 11887 rpcz_store.cc:276] 0113 08:59:22.734583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.971739 (+1237156us) service_pool.cc:225] Handling call
0113 08:59:24.089104 (+117365us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.832401 (+743297us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:24.832600 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:23.713128 14318 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:24.842475 12040 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:24.843502 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2217) took 1242 ms. Trace:
I20250113 08:59:24.846606 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1695) took 1189 ms. Trace:
I20250113 08:59:25.104830 12035 rpcz_store.cc:276] 0113 08:59:23.657528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.632086 (+974558us) service_pool.cc:225] Handling call
0113 08:59:24.846525 (+214439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.104589 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1697) took 1333 ms. Trace:
I20250113 08:59:24.089437 14805 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:24.766639 14488 consensus_meta.cc:287] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.522s user 0.014s sys 0.000s
I20250113 08:59:24.839361 11892 rpcz_store.cc:276] 0113 08:59:22.059979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.981346 (+1921367us) service_pool.cc:225] Handling call
0113 08:59:23.981705 (+ 359us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.296619 (+314914us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:24.296662 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:24.296724 (+ 62us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:24.296772 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:24.296863 (+ 91us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:24.611773 (+314910us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":527484,"raft.queue_time_us":118409,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":55157,"thread_start_us":314396,"threads_started":1}
I20250113 08:59:24.519809 14522 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:24.655787 14759 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:23.979635 14770 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:23.750730 14827 consensus_queue.cc:260] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:24.773084 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2258) took 2698 ms. Trace:
I20250113 08:59:24.765731 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 622) took 1347 ms. Trace:
W20250113 08:59:25.010335 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.40645e+06us
I20250113 08:59:24.958822 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:25.097270 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 623) took 1684 ms (client timeout 1882 ms). Trace:
W20250113 08:59:24.849567 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 45301.5us
I20250113 08:59:25.084200 11961 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:25.135273 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2224) took 1128 ms. Trace:
I20250113 08:59:25.135735 11961 rpcz_store.cc:276] 0113 08:59:24.007141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.737225 (+730084us) service_pool.cc:225] Handling call
0113 08:59:25.135194 (+397969us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:25.008028 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.033s sys 0.051s
W20250113 08:59:25.137215 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.036s user 0.038s sys 0.052s
I20250113 08:59:25.086611 12039 rpcz_store.cc:276] 0113 08:59:23.710556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.763999 (+1053443us) service_pool.cc:225] Handling call
0113 08:59:25.086179 (+322180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:24.607913 12006 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:25.140511 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 532851us
I20250113 08:59:25.140671 11858 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:24.965939 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.142375 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1692) took 1552 ms. Trace:
I20250113 08:59:25.045883 11968 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:25.142454 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:25.142964 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2214) took 1610 ms (client timeout 2092 ms). Trace:
I20250113 08:59:25.143249 12034 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:25.097950 11891 rpcz_store.cc:276] 0113 08:59:22.730703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.971005 (+1240302us) service_pool.cc:225] Handling call
0113 08:59:24.087670 (+116665us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.613099 (+525429us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:24.613886 (+ 787us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.083503 14887 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:25.098138 12041 rpcz_store.cc:276] 0113 08:59:22.686087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.526901 (+840814us) service_pool.cc:225] Handling call
0113 08:59:23.527213 (+ 312us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.527422 (+ 209us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:23.539809 (+ 12387us) log.cc:844] Serialized 14 byte log entry
0113 08:59:24.825942 (+1286133us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:24.826040 (+ 98us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:24.826120 (+ 80us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:24.826220 (+ 100us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:24.826393 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67214,"prepare.queue_time_us":12635,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":69322,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":13028,"thread_start_us":1218413,"threads_started":1}
I20250113 08:59:25.086753 14880 consensus_queue.cc:260] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:24.998144 14885 log.cc:926] Time spent T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20250113 08:59:24.829861 14527 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:24.836014 14798 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:24.765318 14807 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.322s
W20250113 08:59:24.962076 14905 log.cc:926] Time spent T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.233s user 0.000s sys 0.000s
I20250113 08:59:25.105793 12040 rpcz_store.cc:276] 0113 08:59:23.771573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.749586 (+978013us) service_pool.cc:225] Handling call
0113 08:59:25.104522 (+354936us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:25.106153 12008 connection.cc:232] RPC call timeout handler was delayed by -1.3415962400000581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1882s for a response
I20250113 08:59:25.104719 11965 rpcz_store.cc:276] 0113 08:59:23.600993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.996140 (+395147us) service_pool.cc:225] Handling call
0113 08:59:24.843429 (+847289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.102650 11963 rpcz_store.cc:276] 0113 08:59:23.538823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.986726 (+447903us) service_pool.cc:225] Handling call
0113 08:59:24.833434 (+846708us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.103207 12038 raft_consensus.cc:2388] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 4: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:25.108171 11959 raft_consensus.cc:2388] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:24.844188 14760 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:25.008735 14715 consensus_meta.cc:287] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.492s user 0.006s sys 0.006s
I20250113 08:59:24.844719 14727 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:25.220659 14900 log.cc:926] Time spent T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20250113 08:59:25.221572 12726 raft_consensus.cc:2405] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:25.073096 14889 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.975s user 0.024s sys 0.036s
I20250113 08:59:25.222759 14406 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:25.223282 14406 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:25.096894 14865 consensus_meta.cc:287] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.559s user 0.011s sys 0.002s
I20250113 08:59:25.224501 14865 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:25.226106 14903 log.cc:926] Time spent T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
W20250113 08:59:25.085809 14547 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.017s sys 0.032s
I20250113 08:59:25.226727 14865 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:25.226899 14547 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.353s user 0.017s sys 0.033s
I20250113 08:59:25.049609 14860 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:24.899978 14609 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 08:59:25.227670 14860 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:25.228114 14860 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:25.228448 14860 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:24.843925 14907 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:25.106299 14822 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.229619 14907 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.116753 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:25.108580 14779 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.073s sys 0.089s
I20250113 08:59:24.970417 14560 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
W20250113 08:59:25.048722 12037 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.949s user 0.064s sys 0.151s
I20250113 08:59:25.130079 11886 raft_consensus.cc:2405] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:25.126778 14647 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20250113 08:59:25.111431 14879 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:25.143445 11968 rpcz_store.cc:269] 0113 08:59:23.532597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.094692 (+562095us) service_pool.cc:225] Handling call
0113 08:59:25.142895 (+1048203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.145012 12033 rpcz_store.cc:276] 0113 08:59:23.590952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.832810 (+1241858us) service_pool.cc:225] Handling call
0113 08:59:25.142308 (+309498us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:25.179476 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.09069e+06us
I20250113 08:59:25.179697 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:25.150525 11931 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:25.185631 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 525901us
I20250113 08:59:25.186501 12034 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 8.
W20250113 08:59:25.132611 14344 consensus_meta.cc:287] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.038s user 0.014s sys 0.000s
I20250113 08:59:24.850659 14831 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.097947 14561 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:24.833047 14792 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.332s
I20250113 08:59:25.212858 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1700) took 1215 ms. Trace:
I20250113 08:59:25.208258 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.101449 14723 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:25.140349 12720 rpcz_store.cc:269] 0113 08:59:23.412611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.421104 (+ 8493us) service_pool.cc:225] Handling call
0113 08:59:25.086452 (+1665348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.219740 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:25.219452 14760 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:25.220017 12040 raft_consensus.cc:1235] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250113 08:59:25.220499 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:24.846252 14881 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:25.181860 14488 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:25.343217 14318 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:25.343528 14488 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:25.344238 14488 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:25.343911 14318 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:25.130836 11932 connection.cc:232] RPC call timeout handler was delayed by -1.9629650819999824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20250113 08:59:25.240326 12037 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.141s user 0.064s sys 0.156s
I20250113 08:59:25.243046 14822 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:25.243213 14911 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:25.240572 14814 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:25.196316 14527 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:25.389520 14527 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:25.389607 14791 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:25.390983 11890 raft_consensus.cc:2363] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4.
I20250113 08:59:25.391932 14791 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:25.392928 12035 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:25.354454 14729 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:25.128964 12717 rpcz_store.cc:276] 0113 08:59:23.410838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.413095 (+ 2257us) service_pool.cc:225] Handling call
0113 08:59:24.757840 (+1344745us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.394657 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:25.395408 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 846) took 1068 ms. Trace:
I20250113 08:59:25.395720 12717 rpcz_store.cc:276] 0113 08:59:24.327381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.394365 (+1066984us) service_pool.cc:225] Handling call
0113 08:59:25.395353 (+ 988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.396595 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:25.346204 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2250) took 1986 ms. Trace:
I20250113 08:59:25.369977 14881 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:25.397416 11964 rpcz_store.cc:276] 0113 08:59:23.359600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.359943 (+ 343us) service_pool.cc:225] Handling call
0113 08:59:23.382750 (+ 22807us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.092532 (+709782us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:24.092564 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:24.092611 (+ 47us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:24.092654 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.345966 (+1253312us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b540006e0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:25.346012 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:25.346144 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1631900032}
I20250113 08:59:25.398876 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:25.398734 14860 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.399490 11964 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:25.399996 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 482) took 1304 ms. Trace:
I20250113 08:59:25.335354 14757 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:25.400272 11964 rpcz_store.cc:276] 0113 08:59:24.095267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.398630 (+1303363us) service_pool.cc:225] Handling call
0113 08:59:25.399946 (+ 1316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.400302 14733 consensus_queue.cc:260] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:25.253314 14879 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:25.347400 11856 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:25.342578 14907 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:25.401541 14733 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:25.187027 14887 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:25.402042 14770 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:25.402172 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 54864.6us
I20250113 08:59:25.402994 11856 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:25.404074 11856 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:25.347689 12038 rpcz_store.cc:276] 0113 08:59:23.997644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.833388 (+835744us) service_pool.cc:225] Handling call
0113 08:59:25.212801 (+379413us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.348948 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1688) took 1806 ms. Trace:
I20250113 08:59:25.405830 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.405897 12034 rpcz_store.cc:276] 0113 08:59:23.542866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.142148 (+1599282us) service_pool.cc:225] Handling call
0113 08:59:25.348868 (+206720us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.406862 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.407135 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.407485 12034 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:25.407804 12038 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 8.
I20250113 08:59:25.408396 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.349750 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2225) took 1186 ms. Trace:
I20250113 08:59:25.347257 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 471) took 1562 ms. Trace:
I20250113 08:59:25.356523 12724 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:25.369398 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 481) took 1274 ms. Trace:
W20250113 08:59:25.353649 11967 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.006s sys 0.016s
W20250113 08:59:25.353533 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.011s sys 0.010s
I20250113 08:59:25.253391 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:25.246258 11858 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:25.232735 14766 consensus_meta.cc:287] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.826s user 0.010s sys 0.004s
I20250113 08:59:25.372392 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1685) took 1836 ms. Trace:
W20250113 08:59:25.239827 14779 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.921s user 0.074s sys 0.089s
W20250113 08:59:25.374432 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 232883us
I20250113 08:59:25.373087 12039 raft_consensus.cc:2388] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:25.266711 14892 log.cc:926] Time spent T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250113 08:59:25.126950 12036 rpcz_store.cc:276] 0113 08:59:22.075103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.513967 (+1438864us) service_pool.cc:225] Handling call
0113 08:59:23.546460 (+ 32493us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.763117 (+1216657us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:24.773021 (+ 9904us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:25.369546 12686 leader_election.cc:336] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.096s (SENT)
I20250113 08:59:25.386713 14845 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:25.387082 14561 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:25.388208 14723 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:25.380560 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:25.393594 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1686) took 1857 ms. Trace:
W20250113 08:59:25.346597 11932 connection.cc:232] RPC call timeout handler was delayed by -1.679819352000095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3174s for a response
W20250113 08:59:25.348735 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 209187us
I20250113 08:59:25.600154 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 565) took 5781 ms. Trace:
I20250113 08:59:25.408556 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.625947 12042 rpcz_store.cc:276] 0113 08:59:19.818424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.171470 (+3353046us) service_pool.cc:225] Handling call
0113 08:59:23.190717 (+ 19247us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.830838 (+1640121us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.599761 (+768923us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b540002dfd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:25.600039 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2251494272}
I20250113 08:59:25.627137 12038 raft_consensus.cc:2405] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250113 08:59:25.458285 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 653616us
W20250113 08:59:25.629501 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.867s user 0.024s sys 0.014s
I20250113 08:59:25.629354 14792 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:25.633924 14766 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.635855 11966 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:25.474989 14814 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.401130 14860 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:25.468276 14822 raft_consensus.cc:2758] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250113 08:59:25.374734 14912 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:25.495045 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 86116.1us
I20250113 08:59:25.671237 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:25.639031 12724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 837) took 1674 ms (client timeout 1874 ms). Trace:
I20250113 08:59:25.457238 14920 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:59:25.374276 14911 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:25.639647 11968 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
W20250113 08:59:25.623371 12718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 759) took 2473 ms (client timeout 1882 ms). Trace:
I20250113 08:59:25.371289 14609 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:25.636494 11931 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:25.689496 14831 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:25.689796 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 53417.2us
I20250113 08:59:25.689977 14767 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:25.690510 14767 raft_consensus.cc:695] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:25.691406 14767 consensus_queue.cc:237] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:25.692126 11931 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:25.646051 11965 rpcz_store.cc:276] 0113 08:59:23.785029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.219476 (+1434447us) service_pool.cc:225] Handling call
0113 08:59:25.347198 (+127722us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:25.623884 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 347488us
I20250113 08:59:25.652808 14882 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:25.640605 11967 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.878s user 0.013s sys 0.026s
W20250113 08:59:25.648541 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 926298us
I20250113 08:59:25.625996 11963 rpcz_store.cc:276] 0113 08:59:24.095029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.220218 (+1125189us) service_pool.cc:225] Handling call
0113 08:59:25.369335 (+149117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:25.470057 11889 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1835) took 2298 ms (client timeout 2160 ms). Trace:
I20250113 08:59:25.649385 12040 rpcz_store.cc:276] 0113 08:59:23.535693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.212005 (+1676312us) service_pool.cc:225] Handling call
0113 08:59:25.219795 (+ 7790us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:25.372084 (+152289us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.372312 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.412034 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:25.650516 11932 connection.cc:232] RPC call timeout handler was delayed by -1.57541935200004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.329s for a response
W20250113 08:59:25.698212 11889 rpcz_store.cc:269] 0113 08:59:23.171992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.209735 (+ 37743us) service_pool.cc:225] Handling call
0113 08:59:25.469794 (+2260059us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54000d8754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:25.469981 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3261644160}
I20250113 08:59:25.649164 12035 rpcz_store.cc:276] 0113 08:59:23.535976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.376279 (+1840303us) service_pool.cc:225] Handling call
0113 08:59:25.392754 (+ 16475us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:25.393348 (+ 594us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.393536 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.652473 14724 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:25.653239 12033 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250113 08:59:25.640421 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 270902us
I20250113 08:59:25.697527 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2249) took 2340 ms. Trace:
I20250113 08:59:25.726195 11932 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:25.727243 11934 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:25.727524 11931 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:25.728207 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 571) took 3012 ms. Trace:
I20250113 08:59:25.725181 11964 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:25.729357 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 474) took 1857 ms (client timeout 2091 ms). Trace:
W20250113 08:59:25.729902 11964 rpcz_store.cc:269] 0113 08:59:23.872046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.401062 (+1529016us) service_pool.cc:225] Handling call
0113 08:59:25.729238 (+328176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.728580 12041 rpcz_store.cc:276] 0113 08:59:22.715374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.195984 (+2480610us) service_pool.cc:225] Handling call
0113 08:59:25.228449 (+ 32465us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:25.228726 (+ 277us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:25.474607 (+245881us) log.cc:844] Serialized 14 byte log entry
0113 08:59:25.727729 (+253122us) log.cc:844] Serialized 14 byte log entry
0113 08:59:25.727886 (+ 157us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:25.727927 (+ 41us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:25.727976 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.728036 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:25.728153 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27336,"prepare.queue_time_us":458581,"thread_start_us":470616,"threads_started":2}
I20250113 08:59:25.732180 12034 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20250113 08:59:25.682794 14723 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:25.680845 14757 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20250113 08:59:25.686240 14778 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20250113 08:59:25.680487 14798 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.679922 14912 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:25.625423 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 379282us
I20250113 08:59:25.647133 11961 rpcz_store.cc:276] 0113 08:59:24.163780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.179400 (+1015620us) service_pool.cc:225] Handling call
0113 08:59:25.349676 (+170276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.693456 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:25.682866 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2220) took 1929 ms (client timeout 2171 ms). Trace:
I20250113 08:59:25.725271 11967 rpcz_store.cc:276] 0113 08:59:23.357584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.357769 (+ 185us) service_pool.cc:225] Handling call
0113 08:59:23.382366 (+ 24597us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:23.382651 (+ 285us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:23.537385 (+154734us) log.cc:844] Serialized 14 byte log entry
0113 08:59:25.622479 (+2085094us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b18003a8d60. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d1204 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:59:25.697130 (+ 74651us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:25.697186 (+ 56us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:25.697270 (+ 84us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.697358 (+ 88us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:25.697470 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3800,"prepare.queue_time_us":250949,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":345701,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":5362,"spinlock_wait_cycles":45430272,"thread_start_us":2297827,"threads_started":3,"wal-append.queue_time_us":1575122}
I20250113 08:59:25.737418 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.111441 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.112093 11963 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:25.737298 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:25.664026 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.783782 12036 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:25.667179 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 757) took 5282 ms. Trace:
I20250113 08:59:25.442171 14560 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:25.671926 12724 rpcz_store.cc:269] 0113 08:59:23.964844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.775803 (+810959us) service_pool.cc:225] Handling call
0113 08:59:25.638937 (+863134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.666843 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:25.927403 14847 raft_consensus.cc:3049] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:26.152998 14847 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:26.153185 12039 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:25.681964 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.155081 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.155967 12042 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:26.156339 12724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 847) took 1828 ms (client timeout 2147 ms). Trace:
I20250113 08:59:26.156033 14847 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:25.685740 12718 rpcz_store.cc:269] 0113 08:59:23.149671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.239333 (+ 89662us) service_pool.cc:225] Handling call
0113 08:59:25.623046 (+2383713us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54006cb954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:25.623263 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665435392}
I20250113 08:59:25.548476 14791 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:25.390553 14790 consensus_meta.cc:287] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.998s user 0.014s sys 0.000s
I20250113 08:59:25.781462 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.121685 14865 raft_consensus.cc:3049] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
W20250113 08:59:25.808527 12686 connection.cc:232] RPC call timeout handler was delayed by -1.7280221100000972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
W20250113 08:59:25.693166 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.81946e+06us
I20250113 08:59:25.804647 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:26.116421 11965 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2222) took 2284 ms (client timeout 2147 ms). Trace:
I20250113 08:59:25.796541 14766 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.116478 14664 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:26.103822 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 369703us
I20250113 08:59:26.129045 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:25.781513 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.116977 14717 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:25.803959 12684 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:25.744961 14724 raft_consensus.cc:695] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:25.685511 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 848) took 1357 ms (client timeout 1525 ms). Trace:
I20250113 08:59:26.193535 11960 raft_consensus.cc:2405] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:26.193934 12720 rpcz_store.cc:269] 0113 08:59:24.328019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.345991 (+1017972us) service_pool.cc:225] Handling call
0113 08:59:25.682854 (+336863us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.195770 12034 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 10.
I20250113 08:59:26.195881 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:25.801420 14731 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:26.197456 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1689) took 2638 ms. Trace:
I20250113 08:59:26.197861 12033 rpcz_store.cc:276] 0113 08:59:23.543133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.375461 (+1832328us) service_pool.cc:225] Handling call
0113 08:59:25.468745 (+ 93284us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:25.726168 (+257423us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.180798 (+454630us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001f18d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:26.181191 (+ 393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":423465216}
I20250113 08:59:26.197957 14731 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:26.199232 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2212) took 2684 ms (client timeout 1603 ms). Trace:
I20250113 08:59:26.199482 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.199600 14805 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:26.199695 11962 rpcz_store.cc:269] 0113 08:59:23.514450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.517508 (+ 3058us) service_pool.cc:225] Handling call
0113 08:59:26.198772 (+2681264us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b540028e6d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.198958 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1098824064}
I20250113 08:59:26.200431 12033 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:25.802201 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.197417 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.134315 11967 raft_consensus.cc:2388] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:25.946427 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 755) took 5562 ms. Trace:
I20250113 08:59:25.945084 12723 raft_consensus.cc:1270] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250113 08:59:25.771588 14887 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:25.728574 14883 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:26.203891 12721 rpcz_store.cc:276] 0113 08:59:20.384188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.730724 (+2346536us) service_pool.cc:225] Handling call
0113 08:59:22.813572 (+ 82848us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.775732 (+1962160us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.946098 (+1170366us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b540068ddd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:25.946340 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1379622144}
I20250113 08:59:26.177716 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:25.696869 14882 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:25.800570 14904 log.cc:926] Time spent T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.621s user 0.004s sys 0.017s
I20250113 08:59:25.794610 14879 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:26.115193 14729 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:25.402060 14907 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:25.791218 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 624) took 2376 ms. Trace:
W20250113 08:59:26.216014 11890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1823) took 4157 ms (client timeout 2134 ms). Trace:
I20250113 08:59:26.216141 14907 raft_consensus.cc:695] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:25.793221 14488 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.218151 14912 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.217968 14907 consensus_queue.cc:237] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:26.108343 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 381937us
W20250113 08:59:26.109062 11968 rpcz_store.cc:269] 0113 08:59:23.754018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.253052 (+1499034us) service_pool.cc:225] Handling call
0113 08:59:25.682806 (+429754us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:25.734961 14845 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:26.145463 12725 rpcz_store.cc:276] 0113 08:59:20.384830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.106744 (+2721914us) service_pool.cc:225] Handling call
0113 08:59:23.171331 (+ 64587us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:23.178585 (+ 7254us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:23.178614 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:23.183378 (+ 4764us) log.cc:844] Serialized 26 byte log entry
0113 08:59:23.502936 (+319558us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:23.525645 (+ 22709us) log.cc:844] Serialized 14 byte log entry
0113 08:59:25.650995 (+2125350us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b18003f2da0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc33364893a 00007fc33cb04be9 00007fc333509ad0 00007fc3336ce304 00007fc3336cdbf5 00007fc3366013d0 00007fc336613666 00007fc3366241d7 00007fc336624158
0113 08:59:25.651299 (+ 304us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.651411 (+ 112us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:25.651478 (+ 67us) raft_consensus.cc:1680] finished
0113 08:59:25.666908 (+ 15430us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:25.667093 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116473,"prepare.queue_time_us":22689,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":3719,"raft.queue_time_us":2145658,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":8183,"spinlock_wait_cycles":105300352,"thread_start_us":298298,"threads_started":1,"wal-append.queue_time_us":329828}
I20250113 08:59:26.148468 14798 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:25.494139 12717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 844) took 1156 ms. Trace:
I20250113 08:59:26.230670 11964 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:26.156776 12724 rpcz_store.cc:269] 0113 08:59:24.327714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.149939 (+1822225us) service_pool.cc:225] Handling call
0113 08:59:26.156236 (+ 6297us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:25.819918 14906 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.624s user 0.003s sys 0.008s
I20250113 08:59:26.165104 12038 raft_consensus.cc:2388] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:26.191944 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 484) took 2096 ms (client timeout 2106 ms). Trace:
I20250113 08:59:26.192906 12040 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:26.178659 11886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 577) took 4158 ms (client timeout 2524 ms). Trace:
I20250113 08:59:25.801208 14527 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:26.231981 11961 rpcz_store.cc:269] 0113 08:59:24.095753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.128275 (+2032522us) service_pool.cc:225] Handling call
0113 08:59:26.191858 (+ 63583us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.200716 14805 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.203431 12723 pending_rounds.cc:77] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting all ops after (but not including) 5
I20250113 08:59:25.943250 14318 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.202981 12035 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
W20250113 08:59:25.944334 12726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 746) took 8095 ms (client timeout 1592 ms). Trace:
W20250113 08:59:26.189237 11965 rpcz_store.cc:269] 0113 08:59:23.832026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.693134 (+1861108us) service_pool.cc:225] Handling call
0113 08:59:26.114304 (+421170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.176895 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1824) took 4118 ms. Trace:
I20250113 08:59:26.204799 14883 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:25.955955 14760 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:26.205935 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2219) took 2493 ms (client timeout 2450 ms). Trace:
W20250113 08:59:26.238277 11967 rpcz_store.cc:269] 0113 08:59:23.712198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.110449 (+2398251us) service_pool.cc:225] Handling call
0113 08:59:26.205110 (+ 94661us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.197067 12720 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:26.170603 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2247) took 2836 ms (client timeout 2235 ms). Trace:
I20250113 08:59:26.209779 14724 consensus_queue.cc:237] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.036834 14911 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:26.189041 14865 raft_consensus.cc:738] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:25.966084 14795 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:26.094537 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 367132us
I20250113 08:59:26.220444 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:25.948328 14814 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.220793 14887 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:26.243310 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 479) took 2148 ms (client timeout 2003 ms). Trace:
W20250113 08:59:26.219048 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2246) took 2885 ms (client timeout 1660 ms). Trace:
I20250113 08:59:25.938093 14406 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:26.216496 11890 rpcz_store.cc:269] 0113 08:59:22.058197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.130353 (+1072156us) service_pool.cc:225] Handling call
0113 08:59:26.215478 (+3085125us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b540026b954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.215783 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4940383488}
I20250113 08:59:26.249313 11934 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:26.249653 14527 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:26.221951 14621 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:26.250964 14621 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.251159 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.251469 14621 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:26.251910 11934 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:26.252367 11967 raft_consensus.cc:2405] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:26.254307 14766 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:26.254832 14766 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.254966 14664 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:26.255254 14766 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:26.247565 11966 rpcz_store.cc:269] 0113 08:59:23.333915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.334032 (+ 117us) service_pool.cc:225] Handling call
0113 08:59:26.218603 (+2884571us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54001bdad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.218827 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5056296832}
I20250113 08:59:26.223021 14488 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.223162 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:26.224104 14791 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.215924 12719 rpcz_store.cc:276] 0113 08:59:23.415235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.421162 (+ 5927us) service_pool.cc:225] Handling call
0113 08:59:25.695944 (+2274782us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:59:25.765407 (+ 69463us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:25.790713 (+ 25306us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:25.790750 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:25.790803 (+ 53us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:25.790847 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:25.790956 (+ 109us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:25.791111 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33467776}
W20250113 08:59:26.220992 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 417028us
I20250113 08:59:26.228260 14845 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.229933 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:26.150451 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 463) took 2436 ms (client timeout 2354 ms). Trace:
I20250113 08:59:26.182679 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2252) took 4162 ms. Trace:
I20250113 08:59:26.043424 14827 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:26.260929 14845 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:26.261600 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 464) took 2546 ms (client timeout 2136 ms). Trace:
I20250113 08:59:26.262329 12684 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:26.232192 11886 rpcz_store.cc:269] 0113 08:59:22.019836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:22.020082 (+ 246us) service_pool.cc:225] Handling call
0113 08:59:26.177982 (+4157900us) spinlock_profiling.cc:243] Waited 3.07 s on lock 0x7b54000a3f54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.178314 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7054836096}
W20250113 08:59:26.265429 12686 connection.cc:232] RPC call timeout handler was delayed by -1.6425221100000726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23220000000000002s for a response
I20250113 08:59:26.265614 14798 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250113 08:59:26.259966 12033 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
W20250113 08:59:26.259915 11963 rpcz_store.cc:269] 0113 08:59:23.714475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.698219 (+1983744us) service_pool.cc:225] Handling call
0113 08:59:26.144130 (+445911us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.236372 12723 pending_rounds.cc:99] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20250113 08:59:26.234490 12726 rpcz_store.cc:269] 0113 08:59:17.851449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:17.862784 (+ 11335us) service_pool.cc:225] Handling call
0113 08:59:25.944021 (+8081237us) spinlock_profiling.cc:243] Waited 7.36 s on lock 0x7b54006bc954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:25.944222 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16921248128}
I20250113 08:59:26.236538 12034 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:25.738075 14910 log.cc:926] Time spent T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20250113 08:59:26.235268 11888 rpcz_store.cc:276] 0113 08:59:22.058421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.393392 (+1334971us) service_pool.cc:225] Handling call
0113 08:59:23.393778 (+ 386us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.841490 (+1447712us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.176141 (+1334651us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000604d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:26.176608 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3150320384}
I20250113 08:59:26.267048 12723 raft_consensus.cc:2883] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:26.240806 11959 rpcz_store.cc:269] 0113 08:59:23.334447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.334577 (+ 130us) service_pool.cc:225] Handling call
0113 08:59:26.170006 (+2835429us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54000514d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.170331 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3853977344}
I20250113 08:59:26.237282 14883 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:26.244024 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.233142 12717 rpcz_store.cc:276] 0113 08:59:24.311505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.396347 (+1084842us) service_pool.cc:225] Handling call
0113 08:59:25.467604 (+ 71257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.244678 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:25.954011 14880 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:26.248667 12721 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:26.266384 12686 leader_election.cc:336] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.829s (SENT)
I20250113 08:59:26.222270 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.255411 14664 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:26.247421 11964 rpcz_store.cc:269] 0113 08:59:24.094515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.736992 (+1642477us) service_pool.cc:225] Handling call
0113 08:59:26.239092 (+502100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.237692 14912 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:26.237217 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.268581 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.258737 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.239193 12042 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:26.240505 14731 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:26.266098 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:26.262142 11960 rpcz_store.cc:269] 0113 08:59:23.714774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.096465 (+381691us) service_pool.cc:225] Handling call
0113 08:59:26.261284 (+2164819us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54002a4254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.261518 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3633865984}
I20250113 08:59:26.272298 14759 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250113 08:59:26.276283 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:26.273681 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 50728.6us
I20250113 08:59:26.274959 12039 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:26.253050 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 480) took 2158 ms (client timeout 2322 ms). Trace:
I20250113 08:59:26.266141 14798 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.268025 14723 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:26.282141 11967 rpcz_store.cc:269] 0113 08:59:24.094779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.246436 (+2151657us) service_pool.cc:225] Handling call
0113 08:59:26.252945 (+ 6509us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.276863 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.259997 12037 rpcz_store.cc:276] 0113 08:59:22.019700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.191608 (+1171908us) service_pool.cc:225] Handling call
0113 08:59:23.191878 (+ 270us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:24.098960 (+907082us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:24.099266 (+ 306us) log.cc:844] Serialized 14 byte log entry
0113 08:59:25.665845 (+1566579us) log.cc:844] Serialized 14 byte log entry
0113 08:59:25.666087 (+ 242us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:25.666143 (+ 56us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:25.666223 (+ 80us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.182070 (+515847us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b540020dad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:26.182134 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:26.182399 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":617,"prepare.queue_time_us":434826,"prepare.run_cpu_time_us":197,"prepare.run_wall_time_us":201,"raft.queue_time_us":53277,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":2179,"spinlock_wait_cycles":2085637504,"thread_start_us":1565945,"threads_started":2,"wal-append.queue_time_us":1158400}
I20250113 08:59:26.278205 14664 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.282338 11961 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:26.290643 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 477) took 2196 ms (client timeout 2227 ms). Trace:
W20250113 08:59:26.291041 11961 rpcz_store.cc:269] 0113 08:59:24.094007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.240462 (+2146455us) service_pool.cc:225] Handling call
0113 08:59:26.290550 (+ 50088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.277072 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1693) took 2611 ms. Trace:
I20250113 08:59:26.288017 11968 raft_consensus.cc:2405] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:26.284286 14798 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:26.287190 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 843) took 2006 ms. Trace:
I20250113 08:59:26.278363 14882 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:25.886013 14881 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.286475 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.281307 12719 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:26.290210 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.276062 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.293767 12724 rpcz_store.cc:276] 0113 08:59:24.280733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.242759 (+1962026us) service_pool.cc:225] Handling call
0113 08:59:26.287119 (+ 44360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.295470 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.289433 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.292761 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.291749 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.298146 12042 raft_consensus.cc:2388] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:26.285832 14887 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election won for term 8
W20250113 08:59:26.296658 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 483) took 2201 ms (client timeout 2136 ms). Trace:
W20250113 08:59:26.302199 11968 rpcz_store.cc:269] 0113 08:59:24.095505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.232785 (+2137280us) service_pool.cc:225] Handling call
0113 08:59:26.296584 (+ 63799us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:26.300277 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2229) took 1496 ms (client timeout 1830 ms). Trace:
W20250113 08:59:26.304109 11963 rpcz_store.cc:269] 0113 08:59:24.803551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.271172 (+1467621us) service_pool.cc:225] Handling call
0113 08:59:26.300170 (+ 28998us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.303454 14887 raft_consensus.cc:695] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:26.308245 12718 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 7: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:26.305909 12041 raft_consensus.cc:2371] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5. Re-sending same reply.
W20250113 08:59:26.313653 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 47406.9us
I20250113 08:59:26.308912 14847 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:26.301950 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 845) took 1973 ms. Trace:
I20250113 08:59:26.302484 12685 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:26.315747 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.312237 12725 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:26.312165 11966 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:26.318485 11965 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:26.318761 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2253) took 1498 ms (client timeout 1624 ms). Trace:
I20250113 08:59:26.317181 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:26.319231 11966 rpcz_store.cc:269] 0113 08:59:24.820805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.258261 (+1437456us) service_pool.cc:225] Handling call
0113 08:59:26.318708 (+ 60447us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:26.320441 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 61873.6us
I20250113 08:59:26.300139 14912 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:26.321312 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 476) took 2227 ms (client timeout 2810 ms). Trace:
I20250113 08:59:26.311308 14723 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:26.323787 11967 rpcz_store.cc:269] 0113 08:59:24.093640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.296386 (+2202746us) service_pool.cc:225] Handling call
0113 08:59:26.321201 (+ 24815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.293443 12036 rpcz_store.cc:276] 0113 08:59:23.601688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:25.678631 (+2076943us) service_pool.cc:225] Handling call
0113 08:59:25.728633 (+ 50002us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.143937 (+415304us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.213228 (+ 69291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.310855 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.323427 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.326014 12033 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:26.327121 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.327531 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.316601 12719 rpcz_store.cc:276] 0113 08:59:24.326967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.265573 (+1938606us) service_pool.cc:225] Handling call
0113 08:59:26.300622 (+ 35049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.332764 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.333842 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.332337 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.334071 11963 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:26.331696 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:26.334713 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 472) took 2489 ms (client timeout 3238 ms). Trace:
W20250113 08:59:26.353566 11963 rpcz_store.cc:269] 0113 08:59:23.845500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.311920 (+2466420us) service_pool.cc:225] Handling call
0113 08:59:26.334636 (+ 22716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.353876 11968 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:26.354460 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 467) took 2638 ms (client timeout 3327 ms). Trace:
I20250113 08:59:26.354678 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:26.354852 11968 rpcz_store.cc:269] 0113 08:59:23.715618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.310360 (+2594742us) service_pool.cc:225] Handling call
0113 08:59:26.354395 (+ 44035us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.355499 11963 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:26.356050 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.335183 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.335433 11932 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:26.326980 14879 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20250113 08:59:26.358093 11960 raft_consensus.cc:2405] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:26.358286 14729 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election won for term 4
W20250113 08:59:26.358824 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 460) took 2645 ms (client timeout 2920 ms). Trace:
I20250113 08:59:26.358937 14729 raft_consensus.cc:695] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:26.359251 11960 rpcz_store.cc:269] 0113 08:59:23.713604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.292387 (+2578783us) service_pool.cc:225] Handling call
0113 08:59:26.358763 (+ 66376us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.335865 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.360402 11961 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:26.360275 14729 consensus_queue.cc:237] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.337729 12726 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:26.338383 14911 consensus_queue.cc:1046] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250113 08:59:26.341640 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 473) took 2485 ms (client timeout 2374 ms). Trace:
I20250113 08:59:26.360404 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:26.375413 11962 rpcz_store.cc:269] 0113 08:59:23.856940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.332578 (+2475638us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:26.341533 (+ 8955us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:26.335268 14791 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:26.343837 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.301627 14865 consensus_queue.cc:260] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.346326 12008 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:26.360949 14406 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:26.404388 14406 raft_consensus.cc:695] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:26.405741 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 572) took 1027 ms. Trace:
W20250113 08:59:26.361966 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 135565us
I20250113 08:59:26.406157 12037 rpcz_store.cc:276] 0113 08:59:25.378453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.320821 (+942368us) service_pool.cc:225] Handling call
0113 08:59:26.372010 (+ 51189us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.405344 (+ 33334us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:26.405378 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:26.405436 (+ 58us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:26.405487 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.405551 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:26.405665 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.407567 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.409744 14621 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.410524 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.410346 14759 consensus_queue.cc:1035] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250113 08:59:26.336889 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.362569 14887 consensus_queue.cc:237] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.414471 14911 consensus_queue.cc:1046] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:26.416039 14911 consensus_queue.cc:1225] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:26.363971 12725 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:26.405803 14406 consensus_queue.cc:237] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.418077 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.419035 12725 raft_consensus.cc:2363] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7.
I20250113 08:59:26.364204 14881 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:26.366500 11932 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:26.381721 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:26.421973 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 55669us
I20250113 08:59:26.422212 12033 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:26.367290 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 830) took 2493 ms. Trace:
I20250113 08:59:26.367908 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.368165 12038 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:26.424932 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.369482 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:26.427814 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 81736.4us
I20250113 08:59:26.428241 11964 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 3.
I20250113 08:59:26.370028 14805 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.429417 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2231) took 1285 ms. Trace:
I20250113 08:59:26.425091 12723 rpcz_store.cc:276] 0113 08:59:23.874099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.874339 (+ 240us) service_pool.cc:225] Handling call
0113 08:59:25.944516 (+2070177us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.275539 (+331023us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.367002 (+ 91463us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54006bc950. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.367189 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4760322304}
I20250113 08:59:26.354679 14912 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.432907 11964 rpcz_store.cc:276] 0113 08:59:25.143489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.369045 (+1225556us) service_pool.cc:225] Handling call
0113 08:59:26.429072 (+ 60027us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.409803 14717 consensus_queue.cc:1046] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20250113 08:59:26.434123 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.379096 12726 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:56200
I20250113 08:59:26.380865 14790 raft_consensus.cc:3049] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:26.382403 12040 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:26.435213 14790 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:26.382519 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 55720.6us
I20250113 08:59:26.388442 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:26.437371 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:26.439671 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 51525.6us
I20250113 08:59:26.437507 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.403442 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.433606 14881 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:26.435124 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:26.391328 11788 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44470 (request call id 246) took 16280 ms (client timeout 15000 ms). Trace:
I20250113 08:59:26.442406 14791 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250113 08:59:26.475224 14791 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.475607 14791 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Advancing to term 11
I20250113 08:59:26.476162 12039 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:26.478103 12039 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:26.478209 14717 raft_consensus.cc:3049] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:26.478727 14717 raft_consensus.cc:738] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:26.479784 12039 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250113 08:59:26.482223 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.483320 12042 raft_consensus.cc:2388] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 8: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:26.473080 14881 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:26.484687 14881 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.485792 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.427803 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.442410 14783 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:26.486673 11886 raft_consensus.cc:2405] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:26.402714 12721 raft_consensus.cc:2405] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:26.407784 14845 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.424592 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.440181 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2227) took 1705 ms. Trace:
I20250113 08:59:26.393839 11787 catalog_manager.cc:5526] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 5, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 5 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:26.491873 12033 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:26.493351 11888 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:26.429226 12008 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:26.471099 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.408728 12037 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:26.486027 14865 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:26.442631 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2226) took 1788 ms. Trace:
I20250113 08:59:26.496282 11960 rpcz_store.cc:276] 0113 08:59:24.654301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.360064 (+1705763us) service_pool.cc:225] Handling call
0113 08:59:26.442455 (+ 82391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.484927 12042 raft_consensus.cc:1235] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250113 08:59:26.376399 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.498229 11966 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:26.493976 14621 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:26.499274 14767 consensus_queue.cc:1046] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20250113 08:59:26.499950 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.500851 14783 raft_consensus.cc:3049] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:26.500913 14911 consensus_queue.cc:1046] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:59:26.501394 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.501833 14911 consensus_queue.cc:1225] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250113 08:59:26.502472 14911 raft_consensus.cc:3049] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:26.502813 14911 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:26.499320 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.489341 11964 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:26.420629 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.491933 11967 rpcz_store.cc:276] 0113 08:59:24.735588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.332283 (+1596695us) service_pool.cc:225] Handling call
0113 08:59:26.439939 (+107656us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:26.508080 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2221) took 2715 ms (client timeout 3306 ms). Trace:
W20250113 08:59:26.508512 11966 rpcz_store.cc:269] 0113 08:59:23.783191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.325157 (+2541966us) service_pool.cc:225] Handling call
0113 08:59:26.498785 (+173628us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.503823 12038 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
W20250113 08:59:26.512041 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 83395.2us
I20250113 08:59:26.514669 14912 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:26.515146 14912 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:59:26.517375 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 109130us
I20250113 08:59:26.518472 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.520144 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.523887 12034 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:26.523339 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.525626 11961 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:26.526295 12726 raft_consensus.cc:604] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:26.526216 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2230) took 1566 ms (client timeout 2028 ms). Trace:
I20250113 08:59:26.527397 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.527652 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:26.527793 11961 rpcz_store.cc:269] 0113 08:59:24.959782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.301089 (+1341307us) service_pool.cc:225] Handling call
0113 08:59:26.526165 (+225076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.525624 12725 raft_consensus.cc:2388] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:26.501435 14783 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:26.529491 12040 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:26.531319 11962 raft_consensus.cc:2405] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:26.532104 12723 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:26.533107 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.533401 11932 leader_election.cc:400] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:26.533766 11932 leader_election.cc:403] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250113 08:59:26.534226 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.538798 14767 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:26.539593 14767 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250113 08:59:26.542084 12036 raft_consensus.cc:1270] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250113 08:59:26.543118 14601 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:26.543721 14601 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.545696 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.547256 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:26.473680 11788 rpcz_store.cc:269] 0113 08:59:10.111281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:10.166043 (+ 54762us) service_pool.cc:225] Handling call
0113 08:59:26.391015 (+16224972us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:59:18.744679 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:59:18.945992 (+201313us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:59:18.946052 (+ 60us) write_op.cc:435] Acquired schema lock
0113 08:59:18.946066 (+ 14us) tablet.cc:659] Decoding operations
0113 08:59:19.100230 (+154164us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:59:19.100353 (+ 123us) write_op.cc:641] Partition lock acquired for write op
0113 08:59:19.100399 (+ 46us) tablet.cc:682] Acquiring locks for 8 operations
0113 08:59:19.101548 (+ 1149us) tablet.cc:698] Row locks acquired
0113 08:59:19.101632 (+ 84us) write_op.cc:260] PREPARE: finished
0113 08:59:19.101886 (+ 254us) write_op.cc:270] Start()
0113 08:59:19.102065 (+ 179us) write_op.cc:276] Timestamp: P: 1736758759101786 usec, L: 0
0113 08:59:19.102082 (+ 17us) op_driver.cc:348] REPLICATION: starting
0113 08:59:19.274906 (+172824us) log.cc:844] Serialized 3242 byte log entry
0113 08:59:23.075379 (+3800473us) op_driver.cc:464] REPLICATION: finished
0113 08:59:23.207417 (+132038us) write_op.cc:301] APPLY: starting
0113 08:59:23.214815 (+ 7398us) tablet.cc:1364] starting BulkCheckPresence
0113 08:59:23.221340 (+ 6525us) tablet.cc:1367] finished BulkCheckPresence
0113 08:59:23.221521 (+ 181us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:59:23.224177 (+ 2656us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:59:23.224475 (+ 298us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0113 08:59:23.224516 (+ 41us) write_op.cc:312] APPLY: finished
0113 08:59:23.224878 (+ 362us) log.cc:844] Serialized 65 byte log entry
0113 08:59:23.225052 (+ 174us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:23.225184 (+ 132us) write_op.cc:454] Released schema lock
0113 08:59:23.225241 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0113 08:59:23.225676 (+ 435us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:23.225714 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4312401,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":131685,"apply.run_cpu_time_us":3689,"apply.run_wall_time_us":18307,"num_ops":8,"prepare.queue_time_us":1328625,"prepare.run_cpu_time_us":50582,"prepare.run_wall_time_us":3350148,"raft.queue_time_us":922044,"raft.run_cpu_time_us":25495,"raft.run_wall_time_us":108199,"replication_time_us":3973116,"thread_start_us":4158952,"threads_started":4,"wal-append.queue_time_us":1592442,"wal-append.run_cpu_time_us":75969,"wal-append.run_wall_time_us":3342793}]]}
I20250113 08:59:26.548223 11890 raft_consensus.cc:2405] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 7 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:26.549616 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.550623 11889 raft_consensus.cc:2363] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:26.551956 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.530361 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.535364 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.479820 14717 consensus_queue.cc:260] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.521835 14845 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:26.553437 12007 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:26.562664 14911 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.563525 14664 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.563827 14883 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.564437 14911 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:26.521122 14912 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.565364 12717 raft_consensus.cc:2388] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:26.565941 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.567402 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.556633 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.554706 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.571187 11891 raft_consensus.cc:2371] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5. Re-sending same reply.
I20250113 08:59:26.559935 14783 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.572235 12724 raft_consensus.cc:2371] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6. Re-sending same reply.
I20250113 08:59:26.573343 14783 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.573683 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.555313 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.574303 11888 raft_consensus.cc:2371] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6. Re-sending same reply.
I20250113 08:59:26.564467 14717 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.574682 11892 raft_consensus.cc:2363] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4.
I20250113 08:59:26.555959 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.574918 14791 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.575984 11887 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:26.573346 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.577481 11967 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:26.577940 12724 raft_consensus.cc:2405] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:26.578707 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.579210 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.579598 11967 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:26.580610 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.560653 12720 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:26.581347 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.581017 14927 consensus_queue.cc:1046] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:26.582206 12719 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.582218 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.582785 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.582999 14927 consensus_queue.cc:1046] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250113 08:59:26.583637 11960 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:26.581022 12007 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:26.584064 14927 consensus_queue.cc:1035] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:26.584646 14731 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:26.586032 14731 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.586319 14927 raft_consensus.cc:3049] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:26.586464 14731 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.586738 14927 raft_consensus.cc:738] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:26.587769 14927 consensus_queue.cc:260] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.584825 14621 raft_consensus.cc:3049] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:26.589411 14621 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:26.557780 12723 raft_consensus.cc:2388] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:26.590416 14845 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:26.583245 12720 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:26.590400 14621 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.591146 14845 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.590342 14733 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:26.592609 14847 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:26.590579 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.592875 14733 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.593483 14847 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:26.595106 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.596168 12717 raft_consensus.cc:2405] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250113 08:59:26.597913 12008 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:26.597005 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.599290 12008 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:26.600558 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.601289 14647 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:26.601377 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.601747 14647 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.602064 11961 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:59:26.602308 14647 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.601657 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.603368 11892 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 4.
I20250113 08:59:26.600056 14880 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:26.444366 14790 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:26.560860 14882 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.606087 12721 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:26.606194 14805 consensus_queue.cc:1046] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:26.606976 14882 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:26.604898 11856 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:26.607048 14805 consensus_queue.cc:1225] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250113 08:59:26.608090 12721 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69aa379773c047589ae2362850997a47"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:56200
I20250113 08:59:26.597954 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.597288 11934 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:26.584648 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.616428 11960 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:26.620064 11964 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:26.620296 11962 raft_consensus.cc:2388] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:26.631405 11887 raft_consensus.cc:2388] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:26.632303 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.634358 12720 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20250113 08:59:26.635167 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 833) took 2735 ms. Trace:
I20250113 08:59:26.635720 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.636461 14664 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.589097 14927 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:26.637295 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.637697 12721 raft_consensus.cc:604] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:26.638137 12721 raft_consensus.cc:612] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250113 08:59:26.639063 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.640218 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.640882 11964 raft_consensus.cc:2388] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:26.638190 11967 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:26.854760 14798 consensus_meta.cc:287] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.556s user 0.009s sys 0.015s
I20250113 08:59:26.661231 12684 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:26.855435 14798 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.855733 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.856402 14927 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:26.856657 12718 raft_consensus.cc:2363] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 8. Current term is 9.
I20250113 08:59:26.857393 14927 raft_consensus.cc:695] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:26.726114 12037 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:26.643684 11965 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:26.661847 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:26.745424 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 68698.6us
I20250113 08:59:26.730880 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.679141 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.611268 11787 catalog_manager.cc:5526] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:26.684706 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.664047 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.681098 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.656329 11963 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:26.861801 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.857775 14798 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:26.860152 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 199067us
I20250113 08:59:26.867341 11887 raft_consensus.cc:2388] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:26.681624 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.716161 12721 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20250113 08:59:26.862985 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 265806us
I20250113 08:59:26.780031 11961 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 4.
I20250113 08:59:26.685050 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.643167 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.720413 11968 raft_consensus.cc:2405] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:26.659660 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.680109 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.680626 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.854411 14621 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.663373 12007 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:26.675652 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 170001us
I20250113 08:59:26.660555 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.866207 12724 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:26.872107 11960 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:26.686733 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.721921 14766 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.758488 12035 raft_consensus.cc:2405] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:26.868988 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.863665 14601 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:26.873521 14736 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:26.879472 11962 raft_consensus.cc:2405] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250113 08:59:26.880321 14601 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:26.903024 14601 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.642511 14847 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:26.866724 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2260) took 1235 ms (client timeout 1598 ms). Trace:
W20250113 08:59:26.906482 11963 rpcz_store.cc:269] 0113 08:59:25.631078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.354311 (+723233us) service_pool.cc:225] Handling call
0113 08:59:26.866659 (+512348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.907102 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.907716 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.909073 12723 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250113 08:59:26.909479 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.910260 11963 raft_consensus.cc:2388] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:26.868175 11888 raft_consensus.cc:2388] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:26.642203 14845 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.632732 14455 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:26.877039 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 834) took 2969 ms. Trace:
I20250113 08:59:26.912201 11856 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:26.860414 14791 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 11 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.913592 12721 rpcz_store.cc:276] 0113 08:59:23.899627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.638950 (+2739323us) service_pool.cc:225] Handling call
0113 08:59:26.651709 (+ 12759us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.869126 (+217417us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.869335 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.880427 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.914563 11934 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:26.915329 11962 raft_consensus.cc:2405] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:26.689666 14795 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.192s
I20250113 08:59:26.882068 11884 raft_consensus.cc:1270] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250113 08:59:26.882612 14827 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:26.917580 12686 leader_election.cc:336] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.136s (SENT)
I20250113 08:59:26.885524 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.917765 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.917420 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.918414 11887 raft_consensus.cc:2363] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5.
I20250113 08:59:26.918699 12721 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:26.885902 14927 consensus_queue.cc:237] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:26.888610 14934 log.cc:926] Time spent T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.258s user 0.002s sys 0.000s
I20250113 08:59:26.919387 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.885121 12718 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:26.879063 11892 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.920727 11856 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:26.921223 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 835) took 3021 ms. Trace:
I20250113 08:59:26.889308 14664 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:26.890676 11932 leader_election.cc:400] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:26.924865 11932 leader_election.cc:403] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:26.891386 14790 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:26.925789 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.927706 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:26.928496 11890 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:26.922473 11891 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:26.893011 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2279) took 1164 ms. Trace:
I20250113 08:59:26.719388 14881 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.899483 11964 raft_consensus.cc:2405] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:26.899278 11787 catalog_manager.cc:5526] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 4 to 5, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 5 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:59:26.936115 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.635510 12720 rpcz_store.cc:276] 0113 08:59:23.899320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.633481 (+2734161us) service_pool.cc:225] Handling call
0113 08:59:26.634005 (+ 524us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:26.634833 (+ 828us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.635080 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.845088 14880 raft_consensus.cc:695] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:26.873843 11965 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2254) took 2043 ms (client timeout 1616 ms). Trace:
I20250113 08:59:26.942379 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.942379 11886 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
W20250113 08:59:26.944377 11965 rpcz_store.cc:269] 0113 08:59:24.821113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.267100 (+1445987us) service_pool.cc:225] Handling call
0113 08:59:26.864552 (+597452us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.911944 14601 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.910198 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 836) took 2945 ms. Trace:
I20250113 08:59:26.856750 11967 raft_consensus.cc:2405] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:26.910863 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:26.679364 14723 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:26.953461 14723 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:26.659641 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:26.686290 14883 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:26.994756 11890 raft_consensus.cc:2405] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20250113 08:59:26.914268 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.917420 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.917371 14827 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.918380 12038 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.996217 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.884351 12719 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:26.920672 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.921710 12718 rpcz_store.cc:276] 0113 08:59:23.899917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.884248 (+2984331us) service_pool.cc:225] Handling call
0113 08:59:26.884711 (+ 463us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.920912 (+ 36201us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.921147 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.932787 14907 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:26.932957 14664 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:26.934136 11859 leader_election.cc:336] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.603s (SENT)
I20250113 08:59:26.929874 14791 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:26.921993 12686 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:26.934895 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 840) took 2928 ms. Trace:
I20250113 08:59:26.826655 14647 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.006246 14907 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:27.014550 14907 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:27.015930 14647 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.723493 14731 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:27.017524 12042 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250113 08:59:27.017750 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 579) took 1637 ms. Trace:
W20250113 08:59:27.013048 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 95508us
W20250113 08:59:27.020161 12686 leader_election.cc:336] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.354s (SENT)
I20250113 08:59:26.938396 11892 raft_consensus.cc:2405] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:27.006700 14827 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:27.022055 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.997126 14723 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:27.023281 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.023924 11892 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:27.024771 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.026515 11892 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.020861 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.930980 12035 rpcz_store.cc:276] 0113 08:59:25.729359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.309330 (+579971us) service_pool.cc:225] Handling call
0113 08:59:26.892783 (+583453us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b540024ead4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.892937 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":945236096}
I20250113 08:59:26.942282 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:26.942544 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 52044.5us
I20250113 08:59:26.942996 11959 raft_consensus.cc:2363] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8.
I20250113 08:59:26.942898 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.902281 11893 raft_consensus.cc:1235] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:26.945111 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.946229 14887 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:26.946377 12036 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.949963 12723 rpcz_store.cc:276] 0113 08:59:23.964495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.908335 (+2943840us) service_pool.cc:225] Handling call
0113 08:59:26.908705 (+ 370us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.909897 (+ 1192us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.910143 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.950843 14845 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 3 FOLLOWER]: Advancing to term 5
I20250113 08:59:26.951853 11965 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20250113 08:59:26.922420 14455 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:26.953889 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.642534 14733 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.952965 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:26.957482 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2252) took 2137 ms. Trace:
I20250113 08:59:26.957898 11961 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:27.036509 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:27.037043 11932 leader_election.cc:400] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4.
I20250113 08:59:27.037401 11932 leader_election.cc:403] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250113 08:59:27.037806 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:27.038857 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:26.958520 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.959146 11966 raft_consensus.cc:2405] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:27.039996 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:27.040673 12033 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.041275 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:27.041714 12726 raft_consensus.cc:2405] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:27.018270 12041 rpcz_store.cc:276] 0113 08:59:25.380612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.322601 (+941989us) service_pool.cc:225] Handling call
0113 08:59:26.427006 (+104405us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.427411 (+ 405us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:26.524015 (+ 96604us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.008224 (+484209us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.008264 (+ 40us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:27.017425 (+ 9161us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.017518 (+ 93us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.017681 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4731,"prepare.queue_time_us":486166,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":70798,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":16822,"thread_start_us":423705,"threads_started":3,"wal-append.queue_time_us":319638}
I20250113 08:59:27.036266 11967 rpcz_store.cc:276] 0113 08:59:24.820456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.855392 (+2034936us) service_pool.cc:225] Handling call
0113 08:59:26.957411 (+102019us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:26.962862 11856 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:26.963966 14880 consensus_queue.cc:237] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.041880 11966 raft_consensus.cc:2371] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6. Re-sending same reply.
I20250113 08:59:26.965037 11787 catalog_manager.cc:5526] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 4, leader changed from 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131) to 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132). New cstate: current_term: 4 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:59:26.966646 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.950205 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:26.967290 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 304030us
I20250113 08:59:26.971006 12006 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:26.973634 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 103515us
W20250113 08:59:26.974242 14936 log.cc:926] Time spent T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250113 08:59:27.046392 11961 raft_consensus.cc:2463] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:27.047431 12038 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:27.047641 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 470) took 3331 ms (client timeout 3145 ms). Trace:
I20250113 08:59:27.047941 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:26.980019 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.982185 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:26.984910 14766 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.988757 12685 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:26.990057 14881 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:26.989877 14911 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250113 08:59:26.995019 12717 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:26.995587 11960 raft_consensus.cc:2388] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:26.996089 14790 consensus_queue.cc:1035] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:59:26.890424 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:26.996884 11888 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 8.
W20250113 08:59:27.006026 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 96324.1us
I20250113 08:59:27.002046 11887 raft_consensus.cc:2405] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250113 08:59:27.008085 12721 rpcz_store.cc:276] 0113 08:59:23.993742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.914370 (+2920628us) service_pool.cc:225] Handling call
0113 08:59:26.918385 (+ 4015us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.919040 (+ 655us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:26.922007 (+ 2967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.006268 14791 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:27.007540 14883 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.002108 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:27.002665 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:27.034166 14887 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:27.032047 11962 raft_consensus.cc:2405] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:27.032821 12725 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:27.036899 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:26.959177 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 458) took 3246 ms (client timeout 3528 ms). Trace:
I20250113 08:59:26.961315 14912 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.068405 12724 raft_consensus.cc:2405] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20250113 08:59:27.068276 11968 rpcz_store.cc:269] 0113 08:59:23.713006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.355660 (+2642654us) service_pool.cc:225] Handling call
0113 08:59:26.957315 (+601655us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b540006e5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:26.959122 (+ 1807us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":835623808}
I20250113 08:59:27.040005 11891 raft_consensus.cc:2388] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:27.043197 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2211) took 3535 ms. Trace:
I20250113 08:59:27.043740 12037 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:26.975839 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:27.046947 11963 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:27.004591 14664 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:27.073208 14912 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:27.073990 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.074795 11963 raft_consensus.cc:2388] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:27.075593 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:27.076193 11963 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:27.077493 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 88825.2us
I20250113 08:59:27.043817 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.078398 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.078603 11893 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:27.049593 11934 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.079557 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.051734 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.080112 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 768) took 1399 ms. Trace:
I20250113 08:59:27.080581 11967 raft_consensus.cc:2405] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:27.068848 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:27.051091 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:27.050537 14733 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:59:27.081705 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.082413 11968 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:27.082976 12721 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:27.083290 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:27.083869 11968 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.085502 14731 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:27.085968 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 65873.8us
W20250113 08:59:27.086184 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 115554us
I20250113 08:59:27.087239 14455 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:27.069734 12719 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:27.090052 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.090716 11891 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:27.091578 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.053417 12007 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.051868 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:27.058413 11884 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:27.053869 11787 catalog_manager.cc:5526] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 5, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130). New cstate: current_term: 5 leader_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } }
I20250113 08:59:27.059788 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.055109 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.062652 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.062466 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.030117 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.063959 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2271) took 1379 ms. Trace:
I20250113 08:59:27.064954 11931 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.065522 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.066608 12684 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:27.046972 14865 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 3. Current term is 4.
I20250113 08:59:26.958335 12720 raft_consensus.cc:1270] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:27.069675 11965 rpcz_store.cc:276] 0113 08:59:23.499505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.950942 (+3451437us) service_pool.cc:225] Handling call
0113 08:59:26.951565 (+ 623us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:27.034855 (+ 83290us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.035110 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.072444 12724 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:27.061527 14791 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:27.048055 11959 rpcz_store.cc:269] 0113 08:59:23.726814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.367489 (+2640675us) service_pool.cc:225] Handling call
0113 08:59:27.047474 (+679985us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b54002f0654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:27.047583 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1177893632}
I20250113 08:59:27.080164 11887 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:27.055683 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:27.097105 11891 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.103219 11892 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:27.062644 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:27.092489 12723 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:27.098644 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.104553 11886 raft_consensus.cc:2388] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:59:27.110791 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 44317.7us
W20250113 08:59:27.105520 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 60013.4us
I20250113 08:59:27.112030 11934 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.106093 14847 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:27.105341 11787 catalog_manager.cc:5526] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 6, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:27.106599 11888 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:27.107028 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.113590 11959 raft_consensus.cc:2405] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:27.107451 12685 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.108078 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.109602 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 841) took 3115 ms. Trace:
I20250113 08:59:27.109915 12726 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:27.110481 11963 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:27.115953 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.117138 11963 raft_consensus.cc:2405] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:27.117697 11890 raft_consensus.cc:2388] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:27.110707 11932 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.119374 12042 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:27.119648 12041 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:27.119704 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.080725 12718 rpcz_store.cc:276] 0113 08:59:25.681227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.079138 (+1397911us) service_pool.cc:225] Handling call
0113 08:59:27.079465 (+ 327us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.079723 (+ 258us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.079758 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.079800 (+ 42us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:27.079858 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.079916 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.080038 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.122051 12723 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
mode: GRACEFUL
new_leader_uuid: "6ea00811b7d44b49b872ac5c03b91487"
from {username='slave'} at 127.0.0.1:56200
I20250113 08:59:27.122656 12723 raft_consensus.cc:604] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Received request to transfer leadership to TS 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.121977 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:27.123018 12723 raft_consensus.cc:612] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250113 08:59:27.125625 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.126269 11967 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:27.127525 11884 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:27.128477 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.129303 11884 raft_consensus.cc:2405] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:27.129416 12722 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:27.130486 11884 raft_consensus.cc:1270] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
W20250113 08:59:27.130679 12722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 832) took 3255 ms (client timeout 1762 ms). Trace:
W20250113 08:59:27.131196 12722 rpcz_store.cc:269] 0113 08:59:23.898882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:24.772695 (+873813us) service_pool.cc:225] Handling call
0113 08:59:27.130481 (+2357786us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b54007a0654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:27.130598 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5221777792}
I20250113 08:59:27.131971 11965 raft_consensus.cc:1235] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:27.132620 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 461) took 3418 ms. Trace:
I20250113 08:59:27.119104 12721 raft_consensus.cc:2463] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:27.119131 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 765) took 1434 ms. Trace:
I20250113 08:59:27.132925 11965 rpcz_store.cc:276] 0113 08:59:23.713847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.131392 (+3417545us) service_pool.cc:225] Handling call
0113 08:59:27.131767 (+ 375us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.132376 (+ 609us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.132543 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.133554 12724 rpcz_store.cc:276] 0113 08:59:25.666140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.071706 (+1405566us) service_pool.cc:225] Handling call
0113 08:59:27.072074 (+ 368us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.100087 (+ 28013us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.100342 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.130841 11884 pending_rounds.cc:77] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 6
I20250113 08:59:27.135008 11884 pending_rounds.cc:99] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250113 08:59:27.135282 11884 raft_consensus.cc:2883] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:27.146442 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 842) took 2920 ms. Trace:
W20250113 08:59:27.146564 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 58279us
I20250113 08:59:27.135919 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:27.069912 14879 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:27.141175 12037 raft_consensus.cc:2405] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:27.141134 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.070519 14887 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.134253 12686 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.110047 14845 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5.
I20250113 08:59:27.164059 14733 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:27.097190 12039 rpcz_store.cc:276] 0113 08:59:25.684273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.480705 (+796432us) service_pool.cc:225] Handling call
0113 08:59:26.487523 (+ 6818us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:26.608406 (+120883us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:26.866345 (+257939us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.063644 (+197299us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.063697 (+ 53us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:27.063744 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.063804 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.063916 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":308260,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":62,"raft.queue_time_us":113,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":28529,"thread_start_us":418992,"threads_started":2}
I20250113 08:59:27.167418 12039 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250113 08:59:27.168301 11963 raft_consensus.cc:1235] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:27.169677 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2216) took 3580 ms. Trace:
I20250113 08:59:27.170925 11963 rpcz_store.cc:276] 0113 08:59:23.589002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.118385 (+3529383us) service_pool.cc:225] Handling call
0113 08:59:27.167636 (+ 49251us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.169376 (+ 1740us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.169599 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.147030 12726 rpcz_store.cc:276] 0113 08:59:24.214132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.058372 (+2844240us) service_pool.cc:225] Handling call
0113 08:59:27.109608 (+ 51236us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.134569 (+ 24961us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.134774 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.147809 11961 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:27.172644 12035 raft_consensus.cc:1235] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:27.133711 14664 raft_consensus.cc:2743] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 08:59:27.161872 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.149844 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:27.157943 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:27.161542 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.149097 14882 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:27.158306 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:27.156648 12033 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:27.071147 14798 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250113 08:59:27.185535 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 241166us
I20250113 08:59:27.169981 11965 raft_consensus.cc:1235] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:27.149143 12725 raft_consensus.cc:1235] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:27.190963 11959 raft_consensus.cc:1235] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
W20250113 08:59:27.191337 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 79390.9us
I20250113 08:59:27.191720 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2218) took 3535 ms. Trace:
I20250113 08:59:27.145509 11893 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 9.
I20250113 08:59:27.192101 11959 rpcz_store.cc:276] 0113 08:59:23.656042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.120770 (+3464728us) service_pool.cc:225] Handling call
0113 08:59:27.190789 (+ 70019us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.191491 (+ 702us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.191669 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.180976 11787 catalog_manager.cc:5526] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 6, leader changed from 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250113 08:59:27.180159 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 55579.5us
I20250113 08:59:27.193521 12008 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:27.193925 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 764) took 1528 ms. Trace:
I20250113 08:59:27.183859 12038 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:27.194284 12725 rpcz_store.cc:276] 0113 08:59:25.665814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.104048 (+1438234us) service_pool.cc:225] Handling call
0113 08:59:27.143010 (+ 38962us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.193693 (+ 50683us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.193870 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:27.184945 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 77570.2us
I20250113 08:59:27.185693 11887 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 11: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
W20250113 08:59:27.195281 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 101586us
I20250113 08:59:27.186548 11886 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:27.196205 11932 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.183754 14621 raft_consensus.cc:2743] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.199019 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 462) took 3484 ms. Trace:
I20250113 08:59:27.199414 11965 rpcz_store.cc:276] 0113 08:59:23.714165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.148185 (+3434020us) service_pool.cc:225] Handling call
0113 08:59:27.169789 (+ 21604us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.189776 (+ 19987us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.198960 (+ 9184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.201879 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 774) took 1421 ms. Trace:
I20250113 08:59:27.161191 14717 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:27.201607 11934 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.202243 12724 rpcz_store.cc:276] 0113 08:59:25.780723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.134353 (+1353630us) service_pool.cc:225] Handling call
0113 08:59:27.184806 (+ 50453us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.201560 (+ 16754us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.201595 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.201638 (+ 43us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:27.201669 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.201725 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.201825 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.204317 12684 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:27.208356 12725 raft_consensus.cc:1235] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:27.208428 12685 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:27.209450 11859 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.211233 11888 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:27.207918 11964 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:27.214777 12035 raft_consensus.cc:1270] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250113 08:59:27.214928 11963 raft_consensus.cc:1235] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250113 08:59:27.189561 14827 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:27.216392 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 82210.9us
I20250113 08:59:27.217413 11890 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:27.217854 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 465) took 3502 ms. Trace:
I20250113 08:59:27.216395 11932 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:27.188854 14790 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:27.210326 14664 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:27.219199 11787 catalog_manager.cc:5526] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:27.223291 11963 rpcz_store.cc:276] 0113 08:59:23.715068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.172477 (+3457409us) service_pool.cc:225] Handling call
0113 08:59:27.214705 (+ 42228us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.217624 (+ 2919us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.217807 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.231038 12686 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.225579 11961 raft_consensus.cc:1235] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:27.190439 14729 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20250113 08:59:27.228884 11887 raft_consensus.cc:1235] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20250113 08:59:27.227200 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 156375us
I20250113 08:59:27.225035 11934 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.226328 11962 raft_consensus.cc:1235] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250113 08:59:27.224133 11966 raft_consensus.cc:1235] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:27.233223 12037 raft_consensus.cc:1270] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250113 08:59:27.119799 12720 rpcz_store.cc:276] 0113 08:59:23.993988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:26.946741 (+2952753us) service_pool.cc:225] Handling call
0113 08:59:26.957929 (+ 11188us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.109296 (+151367us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.109526 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:27.229219 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 184071us
I20250113 08:59:27.253950 12007 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.216740 12718 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:27.215905 14912 raft_consensus.cc:2798] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:27.244937 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 468) took 3529 ms. Trace:
I20250113 08:59:27.256652 11961 rpcz_store.cc:276] 0113 08:59:23.715904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.173282 (+3457378us) service_pool.cc:225] Handling call
0113 08:59:27.225393 (+ 52111us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.244717 (+ 19324us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.244893 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.247947 11965 raft_consensus.cc:1270] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250113 08:59:27.249189 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2223) took 3287 ms. Trace:
I20250113 08:59:27.257787 11962 rpcz_store.cc:276] 0113 08:59:23.962581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.193635 (+3231054us) service_pool.cc:225] Handling call
0113 08:59:27.226044 (+ 32409us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:27.248908 (+ 22864us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.249132 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.249785 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 469) took 3533 ms. Trace:
I20250113 08:59:27.200291 14791 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:27.259328 11966 rpcz_store.cc:276] 0113 08:59:23.716208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.179632 (+3463424us) service_pool.cc:225] Handling call
0113 08:59:27.212835 (+ 33203us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.249533 (+ 36698us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.249726 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.262698 11959 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 6. Ops: [4.5-4.5]
I20250113 08:59:27.263378 12007 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.265110 11858 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.256027 14882 raft_consensus.cc:695] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:27.266044 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 48593us
I20250113 08:59:27.241966 14717 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250113 08:59:27.267033 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 82053.7us
W20250113 08:59:27.269884 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 69253.3us
I20250113 08:59:27.257607 11965 pending_rounds.cc:77] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting all ops after (but not including) 5
I20250113 08:59:27.241779 14664 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:27.217898 14847 consensus_queue.cc:1035] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20250113 08:59:27.267383 11934 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.270619 11965 pending_rounds.cc:99] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250113 08:59:27.272796 11965 raft_consensus.cc:2883] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:27.274953 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 44054.4us
I20250113 08:59:27.249013 14798 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:27.268074 11787 catalog_manager.cc:5526] T fcc31099ea8040b7bfbfe7d779148ca5 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250113 08:59:27.226810 14736 consensus_queue.cc:1035] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20250113 08:59:27.283802 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 478) took 3189 ms. Trace:
I20250113 08:59:27.261585 14621 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:27.239701 14733 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20250113 08:59:27.266470 14941 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.286306 11965 rpcz_store.cc:276] 0113 08:59:24.094269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.203810 (+3109541us) service_pool.cc:225] Handling call
0113 08:59:27.244375 (+ 40565us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.283513 (+ 39138us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.283740 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.273722 14911 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:27.287689 11934 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:27.288761 11962 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 3. Current term is 4. Ops: []
I20250113 08:59:27.289911 11889 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:27.269758 14717 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:27.292294 12686 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:27.275638 14406 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:27.288681 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 76187.6us
I20250113 08:59:27.276279 14664 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.297343 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2258) took 1659 ms. Trace:
I20250113 08:59:27.282204 14790 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:27.291815 12007 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.281739 14880 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:27.295159 12006 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.305804 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2256) took 2346 ms. Trace:
I20250113 08:59:27.306275 11961 rpcz_store.cc:276] 0113 08:59:24.958992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.257466 (+2298474us) service_pool.cc:225] Handling call
0113 08:59:27.290125 (+ 32659us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.305374 (+ 15249us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.305412 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.305483 (+ 71us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:27.305532 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.305604 (+ 72us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.305739 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.303066 11966 rpcz_store.cc:276] 0113 08:59:25.630433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.262485 (+1632052us) service_pool.cc:225] Handling call
0113 08:59:27.286808 (+ 24323us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.289585 (+ 2777us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.289620 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.289672 (+ 52us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:27.289699 (+ 27us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.289755 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.289871 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.304602 11964 raft_consensus.cc:1270] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250113 08:59:27.282131 14912 raft_consensus.cc:695] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:27.313998 12719 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:27.289283 14731 consensus_queue.cc:1046] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20250113 08:59:27.305002 12717 raft_consensus.cc:2463] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:27.319468 11965 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:27.319058 11787 catalog_manager.cc:5526] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } } }
I20250113 08:59:27.320111 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2259) took 1689 ms. Trace:
I20250113 08:59:27.320439 11965 rpcz_store.cc:276] 0113 08:59:25.630798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.288515 (+1657717us) service_pool.cc:225] Handling call
0113 08:59:27.319306 (+ 30791us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.319895 (+ 589us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.320060 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.321431 14795 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.304467 11934 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.325791 12685 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.329452 12036 raft_consensus.cc:2405] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:27.320555 11787 catalog_manager.cc:5526] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 3 to 6, leader changed from 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:59:27.334654 11886 raft_consensus.cc:2405] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:27.337920 12008 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:27.339008 11891 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:27.341737 11931 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:27.342005 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 50290.1us
I20250113 08:59:27.287884 14621 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:27.343379 12042 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 9.
I20250113 08:59:27.337486 14455 raft_consensus.cc:2743] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:27.349525 14907 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:27.350638 14946 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:27.351079 14946 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:27.351439 14946 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:27.351732 14907 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:27.352598 14907 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:27.352946 14907 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:27.290555 14798 raft_consensus.cc:695] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:27.337390 14883 consensus_queue.cc:1046] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20250113 08:59:27.353585 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:27.294512 14717 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.363507 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:27.364439 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:27.364818 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 77370.1us
I20250113 08:59:27.304243 14766 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:27.365077 12725 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:27.359087 11968 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:27.367051 11967 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 7. Ops: []
I20250113 08:59:27.382333 14883 raft_consensus.cc:3049] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:27.418110 11787 catalog_manager.cc:5526] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 4, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 4 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:59:27.384305 14647 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:27.384610 11856 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:27.354846 14911 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.355448 14880 raft_consensus.cc:695] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:27.385490 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 59782.5us
I20250113 08:59:27.385625 14791 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:27.386319 14790 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.453547 12685 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.386070 14882 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.386767 14455 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:27.349573 14944 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:59:27.387435 14766 raft_consensus.cc:695] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:27.358278 14912 consensus_queue.cc:237] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.354099 14798 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.289713 14783 consensus_queue.cc:1035] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20250113 08:59:27.393033 11891 raft_consensus.cc:1235] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:27.338061 14601 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.394193 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2263) took 1062 ms. Trace:
I20250113 08:59:27.355940 14907 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:27.356164 14621 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:27.394358 12686 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:27.444170 11787 catalog_manager.cc:5526] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 2 to 5, leader changed from 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 5 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250113 08:59:27.625494 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 172032us
W20250113 08:59:27.626257 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 231962us
I20250113 08:59:27.626343 14455 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:27.636994 14880 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.637789 14455 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:27.626763 14948 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.232s user 0.001s sys 0.000s
I20250113 08:59:27.629585 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 459) took 3905 ms. Trace:
I20250113 08:59:27.618775 14879 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:27.641690 14879 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:27.438545 14865 raft_consensus.cc:2743] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:27.454231 11963 pending_rounds.cc:77] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting all ops after (but not including) 4
I20250113 08:59:27.642820 11963 pending_rounds.cc:99] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250113 08:59:27.640794 11967 rpcz_store.cc:276] 0113 08:59:23.713332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.126891 (+3413559us) service_pool.cc:225] Handling call
0113 08:59:27.366876 (+239985us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.436248 (+ 69372us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.618110 (+181862us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b5400196250. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:27.618311 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":551000832}
I20250113 08:59:27.643115 11963 raft_consensus.cc:2883] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:27.624691 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 359656us
I20250113 08:59:27.646085 12034 raft_consensus.cc:1235] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
W20250113 08:59:27.646140 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 312620us
I20250113 08:59:27.486958 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 475) took 3468 ms. Trace:
W20250113 08:59:27.606380 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 316831us
I20250113 08:59:27.633870 11968 rpcz_store.cc:276] 0113 08:59:26.309059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.368626 (+1059567us) service_pool.cc:225] Handling call
0113 08:59:27.370374 (+ 1748us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.370637 (+ 263us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.370677 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.370734 (+ 57us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:27.370786 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.370854 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.371000 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:27.649886 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 445624us
W20250113 08:59:27.516140 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 60053.6us
W20250113 08:59:27.498505 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 113991us
I20250113 08:59:27.651487 12007 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:27.636039 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 432946us
I20250113 08:59:27.629319 14954 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader election won for term 9
I20250113 08:59:27.655010 11968 raft_consensus.cc:1235] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:27.654848 12723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 769) took 1948 ms. Trace:
I20250113 08:59:27.648856 11959 rpcz_store.cc:276] 0113 08:59:23.985582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.203256 (+3217674us) service_pool.cc:225] Handling call
0113 08:59:27.243156 (+ 39900us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.263072 (+ 19916us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.453735 (+190663us) spinlock_profiling.cc:243] Waited 6.1 ms on lock 0x7b540039fb54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:27.454018 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14023424}
I20250113 08:59:27.656729 11934 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.653194 11859 leader_election.cc:400] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7.
I20250113 08:59:27.686125 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2264) took 1371 ms. Trace:
I20250113 08:59:27.687860 11859 leader_election.cc:403] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:27.687927 11967 rpcz_store.cc:276] 0113 08:59:26.315256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.643370 (+1328114us) service_pool.cc:225] Handling call
0113 08:59:27.684093 (+ 40723us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.685629 (+ 1536us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.685669 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.685725 (+ 56us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:27.685786 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.685867 (+ 81us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.686045 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.498963 14950 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:27.418658 14883 raft_consensus.cc:738] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:27.485261 14881 raft_consensus.cc:2798] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:27.515066 14406 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.692081 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2268) took 1341 ms. Trace:
I20250113 08:59:27.694204 11856 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.485155 14941 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:27.697890 12041 consensus_meta.cc:287] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.577s user 0.010s sys 0.000s
I20250113 08:59:27.629326 14944 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:27.684823 12723 rpcz_store.cc:276] 0113 08:59:25.695730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.125338 (+1429608us) service_pool.cc:225] Handling call
0113 08:59:27.125702 (+ 364us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.153599 (+ 27897us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.153645 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.163274 (+ 9629us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.237758 (+ 74484us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:27.554921 (+317163us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.555171 (+ 250us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.555279 (+ 108us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:27.643734 (+ 88455us) raft_consensus.cc:1680] finished
0113 08:59:27.643752 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.643905 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41069,"prepare.queue_time_us":55565,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":106305,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":21736,"thread_start_us":358395,"threads_started":2,"wal-append.queue_time_us":122655}
I20250113 08:59:27.524464 14951 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:27.701887 11967 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250113 08:59:27.706207 12725 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 8.
W20250113 08:59:27.708232 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 51593.5us
I20250113 08:59:27.708997 11968 rpcz_store.cc:276] 0113 08:59:26.350318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.649797 (+1299479us) service_pool.cc:225] Handling call
0113 08:59:27.650168 (+ 371us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.691739 (+ 41571us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.692005 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:27.711913 12686 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:27.699246 11787 catalog_manager.cc:5526] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250113 08:59:27.710327 11858 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.714565 12041 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:27.730346 12006 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:27.719025 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2232) took 1582 ms. Trace:
I20250113 08:59:27.714252 14941 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:27.692219 14733 raft_consensus.cc:2798] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:27.690979 14790 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:27.518543 14887 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:27.750093 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 61596.6us
I20250113 08:59:27.720433 14881 raft_consensus.cc:695] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:27.741200 11961 rpcz_store.cc:276] 0113 08:59:26.136595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.307558 (+1170963us) service_pool.cc:225] Handling call
0113 08:59:27.308020 (+ 462us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.308307 (+ 287us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:27.359049 (+ 50742us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.717167 (+358118us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:27.717233 (+ 66us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:27.717308 (+ 75us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.718837 (+ 1529us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.718961 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9117,"prepare.queue_time_us":1344,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"thread_start_us":313435,"threads_started":3}
I20250113 08:59:27.720212 14736 raft_consensus.cc:2798] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:27.722136 14791 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7.
I20250113 08:59:27.719102 14950 raft_consensus.cc:695] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:27.721029 14944 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:27.744479 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 57415.9us
I20250113 08:59:27.721999 14964 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:27.751543 12685 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:27.789161 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 138429us
I20250113 08:59:27.752652 11856 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.792196 11960 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:27.793529 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 140502us
I20250113 08:59:27.793910 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:43866 (request call id 626) took 3697 ms. Trace:
W20250113 08:59:27.794700 14955 log.cc:926] Time spent T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250113 08:59:27.794420 14927 consensus_queue.cc:1046] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:27.794975 14963 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:27.848642 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 107150us
I20250113 08:59:27.795543 14960 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 08:59:27.755887 12719 raft_consensus.cc:1235] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20250113 08:59:27.797224 14766 consensus_queue.cc:237] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.798027 14940 raft_consensus.cc:3049] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:27.798125 14966 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:27.801538 14887 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:27.802153 14959 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250113 08:59:27.801818 14950 consensus_queue.cc:237] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:27.756124 14883 consensus_queue.cc:260] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.803263 14964 consensus_queue.cc:1046] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250113 08:59:27.727659 14951 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:27.804438 14976 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:27.808375 14664 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.809242 14881 consensus_queue.cc:237] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.754350 14790 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.750818 14953 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:27.748178 14941 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:27.810042 14621 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:27.809459 12718 consensus_meta.cc:287] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.532s user 0.005s sys 0.005s
I20250113 08:59:27.885036 14967 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:27.885352 12718 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:27.754377 14946 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:27.811842 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 98547.1us
I20250113 08:59:27.756495 11787 catalog_manager.cc:5526] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 reported cstate change: term changed from 2 to 6, leader changed from 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130) to 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131). New cstate: current_term: 6 leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250113 08:59:27.837656 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 108544us
W20250113 08:59:27.760283 14723 consensus_meta.cc:287] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.737s user 0.010s sys 0.002s
I20250113 08:59:27.756731 12720 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:27.840358 11858 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:27.906834 14723 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.740998 14406 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20250113 08:59:27.752187 11934 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:27.909092 14723 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:27.909106 11885 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:27.844089 12684 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:27.844208 11968 rpcz_store.cc:276] 0113 08:59:24.096146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.722983 (+3626837us) service_pool.cc:225] Handling call
0113 08:59:27.793784 (+ 70801us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:27.848606 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 97147.8us
W20250113 08:59:27.849169 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 153470us
I20250113 08:59:27.911721 12686 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:27.815716 14736 raft_consensus.cc:695] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:27.858016 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 66240.8us
I20250113 08:59:27.912639 14911 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election won for term 8
W20250113 08:59:27.861976 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 140496us
I20250113 08:59:27.913110 14736 consensus_queue.cc:237] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:27.914980 11839 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 08aa17121f5b429b8a32299911c7ef76, TS 5e8883f5f83a47f4b9303dd1da4f59c0 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:27.915382 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 08aa17121f5b429b8a32299911c7ef76, TS 5e8883f5f83a47f4b9303dd1da4f59c0 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:27.913190 14911 raft_consensus.cc:695] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:27.917912 14911 consensus_queue.cc:237] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.920763 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:27.921144 14827 consensus_queue.cc:1046] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:27.864949 11859 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:27.864827 14927 raft_consensus.cc:3049] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:27.922253 14927 raft_consensus.cc:738] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:27.923369 14927 consensus_queue.cc:260] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.924783 14927 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:27.925838 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:27.860925 14940 raft_consensus.cc:738] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:27.863307 11856 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:27.902014 12724 raft_consensus.cc:1235] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:27.901862 14731 raft_consensus.cc:1059] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: attempting to promote NON_VOTER 5f9c33777f6845c2848a2a6102a1c2ff to VOTER
I20250113 08:59:27.885241 14952 consensus_queue.cc:1035] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250113 08:59:27.886417 14978 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election won for term 7
W20250113 08:59:27.909873 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 157753us
I20250113 08:59:27.911495 14985 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:27.921537 11891 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:27.929214 14968 log.cc:926] Time spent T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20250113 08:59:27.951963 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 107988us
I20250113 08:59:27.954438 11886 raft_consensus.cc:1235] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:27.903657 14621 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:27.902721 14883 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:27.930756 14975 log.cc:926] Time spent T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20250113 08:59:27.849713 14954 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:27.962606 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:27.968809 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 52625.7us
I20250113 08:59:27.930529 14865 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:28.005352 12006 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:28.003706 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 92074.8us
W20250113 08:59:28.008210 11839 auto_rebalancer.cc:663] Could not move replica: Illegal state: fc84bb347ec2413e9188b7253b0812ab: source replica 37bd4bed3ee6423c8dac0cef3bf6882a does not have REPLACE attribute set
W20250113 08:59:28.008582 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: fc84bb347ec2413e9188b7253b0812ab: source replica 37bd4bed3ee6423c8dac0cef3bf6882a does not have REPLACE attribute set
I20250113 08:59:27.979123 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 771) took 2227 ms. Trace:
I20250113 08:59:27.969558 14795 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:27.958498 14406 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:28.016112 12722 rpcz_store.cc:276] 0113 08:59:25.733788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.132187 (+1398399us) service_pool.cc:225] Handling call
0113 08:59:27.132531 (+ 344us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.163327 (+ 30796us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:27.200972 (+ 37645us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.549614 (+348642us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.593000 (+ 43386us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.593273 (+ 273us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:27.593932 (+ 659us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.794038 (+200106us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b480005c358. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33b4cd581 00007fc33b4c4677 00007fc33b4c450a 00007fc33b5b89bf 00007fc33b5b8687 00007fc33b5ccb8c 00007fc33b5ccb38 00007fc33b5ccab8 00007fc33b5cca73 00007fc33b5cbd1c 00007fc33a538d16
0113 08:59:27.918511 (+124473us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b18003f2560. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336b2e55 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336cdbf5 00007fc3366013d0 00007fc336613666 00007fc3366241d7 00007fc336624158
0113 08:59:27.957991 (+ 39480us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.958083 (+ 92us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:27.960833 (+ 2750us) raft_consensus.cc:1680] finished
0113 08:59:27.960858 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:27.960988 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20635,"prepare.queue_time_us":106152,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":90199,"raft.run_cpu_time_us":13049,"raft.run_wall_time_us":623146,"spinlock_wait_cycles":81390080,"thread_start_us":396581,"threads_started":3,"wal-append.queue_time_us":482216}
I20250113 08:59:27.859295 14963 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:27.970739 12684 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:28.032696 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 118028us
I20250113 08:59:27.951574 14907 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.010147 14791 raft_consensus.cc:2758] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20250113 08:59:28.047073 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 76107.1us
I20250113 08:59:28.012284 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:28.052511 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 193955us
I20250113 08:59:28.059505 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:28.010748 14733 raft_consensus.cc:695] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:28.060456 11968 raft_consensus.cc:2405] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:28.011096 14865 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:28.032014 14406 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:27.935817 14940 consensus_queue.cc:260] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.084172 14406 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:28.032909 14455 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.089922 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:28.189090 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 169566us
I20250113 08:59:28.201156 11961 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:28.201591 12686 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:28.194376 11934 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:28.113576 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 250472us
I20250113 08:59:28.194705 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1720) took 1079 ms. Trace:
I20250113 08:59:28.212769 12033 rpcz_store.cc:276] 0113 08:59:27.115769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.184695 (+ 68926us) service_pool.cc:225] Handling call
0113 08:59:27.185054 (+ 359us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:27.207853 (+ 22799us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.207889 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:27.229958 (+ 22069us) log.cc:844] Serialized 52 byte log entry
0113 08:59:27.545911 (+315953us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:27.865819 (+319908us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.879440 (+ 13621us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.879520 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.194387 (+314867us) raft_consensus.cc:1680] finished
0113 08:59:28.194419 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:28.194620 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4206,"prepare.queue_time_us":407846,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":7627,"raft.queue_time_us":29726,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":25531,"spinlock_wait_cycles":15421312,"thread_start_us":517812,"threads_started":2,"wal-append.queue_time_us":380306}
I20250113 08:59:28.215946 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 778) took 1663 ms. Trace:
W20250113 08:59:28.216202 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 257848us
I20250113 08:59:28.084618 14991 raft_consensus.cc:3049] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:28.229768 14946 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:28.216483 12726 rpcz_store.cc:276] 0113 08:59:26.552161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.172804 (+620643us) service_pool.cc:225] Handling call
0113 08:59:27.190812 (+ 18008us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.229293 (+ 38481us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.229323 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.250351 (+ 21028us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.668680 (+418329us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:27.668728 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.668820 (+ 92us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:27.928419 (+259599us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:28.203951 (+275532us) raft_consensus.cc:1680] finished
0113 08:59:28.203979 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:28.215868 (+ 11889us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22475,"raft.queue_time_us":31358,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":9253,"spinlock_wait_cycles":79503232,"thread_start_us":402476,"threads_started":1,"wal-append.queue_time_us":537544}
I20250113 08:59:28.222244 14976 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:28.234431 12039 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250113 08:59:28.236290 14973 log.cc:926] Time spent T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.440s user 0.000s sys 0.002s
I20250113 08:59:28.239948 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1979) took 1287 ms. Trace:
I20250113 08:59:28.234201 12686 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:28.241184 11892 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 6. Ops: []
I20250113 08:59:28.187987 14733 consensus_queue.cc:237] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:28.240377 11889 rpcz_store.cc:276] 0113 08:59:26.952695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.049555 (+ 96860us) service_pool.cc:225] Handling call
0113 08:59:28.239689 (+1190134us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b540037e454. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:28.239856 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":440087936}
I20250113 08:59:28.245268 14964 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:28.245733 11859 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:28.188350 14981 raft_consensus.cc:3049] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 LEADER]: Stepping down as leader of term 10
I20250113 08:59:28.246924 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1970) took 1338 ms. Trace:
I20250113 08:59:28.212450 14911 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 08:59:28.248435 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 594) took 1286 ms. Trace:
I20250113 08:59:28.248296 14994 consensus_queue.cc:1046] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.162s
I20250113 08:59:28.283475 11885 rpcz_store.cc:276] 0113 08:59:26.962448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.053518 (+ 91070us) service_pool.cc:225] Handling call
0113 08:59:28.248218 (+1194700us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54003af054. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:28.248377 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1866840448}
I20250113 08:59:28.255174 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2257) took 3282 ms. Trace:
W20250113 08:59:28.255209 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 248358us
I20250113 08:59:28.283079 11892 rpcz_store.cc:276] 0113 08:59:26.908136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.134818 (+226682us) service_pool.cc:225] Handling call
0113 08:59:28.240994 (+1106176us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:28.245680 (+ 4686us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.246672 (+ 992us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540037e450. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:28.246819 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2491773184}
I20250113 08:59:28.291688 11962 rpcz_store.cc:276] 0113 08:59:24.959355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.258824 (+2299469us) service_pool.cc:225] Handling call
0113 08:59:27.288576 (+ 29752us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.290064 (+ 1488us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.241827 (+951763us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b54000f49a0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336599990 00007fc33660cb42 00007fc336608f26 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782
0113 08:59:28.242062 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6572928}
I20250113 08:59:28.066613 14795 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:28.188858 14986 consensus_queue.cc:1046] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
W20250113 08:59:28.337836 14879 consensus_meta.cc:287] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.696s user 0.010s sys 0.002s
W20250113 08:59:28.338109 14997 log.cc:926] Time spent T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20250113 08:59:28.338410 14879 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:28.339879 14944 consensus_meta.cc:287] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.541s user 0.010s sys 0.000s
I20250113 08:59:28.282027 14981 raft_consensus.cc:738] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:28.224653 14991 raft_consensus.cc:738] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:28.047456 14723 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:28.302358 12006 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:28.303035 11839 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7d7522e543294e28ac66b2faf6001cbe, TS 6ea00811b7d44b49b872ac5c03b91487 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:28.253373 14845 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:28.086196 14951 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:28.333803 14998 log.cc:926] Time spent T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20250113 08:59:28.334232 14993 raft_consensus.cc:2743] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:28.280856 14964 consensus_queue.cc:1225] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:28.256880 14621 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:28.297339 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 505608us
I20250113 08:59:28.300206 11961 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:28.340449 14944 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:28.366475 11839 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7d7522e543294e28ac66b2faf6001cbe, TS 6ea00811b7d44b49b872ac5c03b91487 -> TS 5f9c33777f6845c2848a2a6102a1c2ff move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:28.329362 14736 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:28.326663 14946 raft_consensus.cc:2743] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:28.279359 14967 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:28.159893 14954 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.388285 14978 raft_consensus.cc:695] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:28.371081 14723 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:28.300348 14790 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:28.484578 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 404408us
W20250113 08:59:28.480963 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 247183us
I20250113 08:59:28.633535 11858 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:28.376127 14985 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:28.631677 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 397875us
W20250113 08:59:28.419289 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 173652us
I20250113 08:59:28.636621 12686 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:28.597146 14717 consensus_meta.cc:287] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.208s user 0.012s sys 0.001s
W20250113 08:59:28.634927 14962 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.845s user 0.001s sys 0.001s
W20250113 08:59:28.563315 12717 consensus_meta.cc:287] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.834s user 0.011s sys 0.000s
I20250113 08:59:28.375440 14845 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:28.603830 14961 log.cc:926] Time spent T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.802s user 0.000s sys 0.002s
I20250113 08:59:28.637851 14865 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:28.595427 14940 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:28.653461 12006 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:28.634406 14723 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:28.727864 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:28.474545 14879 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:28.474261 14941 consensus_meta.cc:287] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.589s user 0.000s sys 0.010s
I20250113 08:59:28.596776 14956 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:28.665283 11885 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20250113 08:59:28.665681 14974 log.cc:926] Time spent T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.869s user 0.002s sys 0.000s
I20250113 08:59:28.666359 11893 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250113 08:59:28.640194 14795 raft_consensus.cc:695] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:28.663684 14717 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:28.770958 14795 consensus_queue.cc:237] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.772738 14717 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:28.762903 11893 pending_rounds.cc:77] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 6
I20250113 08:59:28.773733 11893 pending_rounds.cc:99] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20250113 08:59:28.773984 11893 raft_consensus.cc:2883] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:28.774053 11885 raft_consensus.cc:1235] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:28.661149 12717 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
W20250113 08:59:28.670295 12718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 785) took 1576 ms (client timeout 1988 ms). Trace:
I20250113 08:59:28.673790 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1848) took 1615 ms. Trace:
I20250113 08:59:28.544414 14999 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:28.775753 12718 rpcz_store.cc:269] 0113 08:59:27.093496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.121712 (+ 28216us) service_pool.cc:225] Handling call
0113 08:59:28.670038 (+1548326us) spinlock_profiling.cc:243] Waited 90.4 ms on lock 0x7b54006b1cd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:28.670187 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207993216}
I20250113 08:59:28.775506 11885 raft_consensus.cc:1270] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:28.690286 12041 raft_consensus.cc:1270] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250113 08:59:28.691009 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1855) took 1612 ms. Trace:
I20250113 08:59:28.691411 12039 pending_rounds.cc:77] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 6
I20250113 08:59:28.778211 12039 pending_rounds.cc:99] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20250113 08:59:28.715502 14455 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:28.693473 14994 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:28.785954 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 79530.9us
W20250113 08:59:28.694325 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 57796.2us
I20250113 08:59:28.694573 14927 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:28.464416 14911 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:28.389122 14991 consensus_queue.cc:260] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.697647 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 597) took 1708 ms. Trace:
I20250113 08:59:28.603631 14406 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.702718 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2228) took 3941 ms. Trace:
W20250113 08:59:28.702826 14977 log.cc:926] Time spent T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.900s user 0.001s sys 0.001s
I20250113 08:59:28.705919 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:28.708511 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 75098.3us
I20250113 08:59:28.811328 11964 rpcz_store.cc:276] 0113 08:59:24.761680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.247353 (+2485673us) service_pool.cc:225] Handling call
0113 08:59:27.265705 (+ 18352us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.309765 (+ 44060us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.702389 (+1392624us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b540028f0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:28.702616 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88592256}
I20250113 08:59:28.812026 11858 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:28.705004 14976 raft_consensus.cc:3049] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:28.710136 14963 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:28.785799 14883 consensus_meta.cc:287] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.781s user 0.005s sys 0.007s
I20250113 08:59:28.711472 14978 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.712616 14967 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:28.707886 12007 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:28.644819 14907 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:28.713099 14995 log.cc:926] Time spent T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.474s user 0.003s sys 0.000s
I20250113 08:59:28.765019 12720 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:28.774143 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1977) took 1822 ms. Trace:
I20250113 08:59:28.775967 11890 rpcz_store.cc:276] 0113 08:59:27.058236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.232251 (+174015us) service_pool.cc:225] Handling call
0113 08:59:27.247698 (+ 15447us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.255607 (+ 7909us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.255629 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.272053 (+ 16424us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.635951 (+363898us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:27.636008 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.636098 (+ 90us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.564183 (+928085us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:28.673441 (+109258us) raft_consensus.cc:1680] finished
0113 08:59:28.673467 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:28.673720 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99125120,"thread_start_us":333686,"threads_started":1,"wal-append.queue_time_us":436191}
I20250113 08:59:28.689736 14729 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:28.777842 11886 rpcz_store.cc:276] 0113 08:59:27.079088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.360640 (+281552us) service_pool.cc:225] Handling call
0113 08:59:27.438193 (+ 77553us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.962572 (+524379us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.690762 (+728190us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540034c6d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:28.690949 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1186644864}
I20250113 08:59:28.691859 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 643) took 1035 ms. Trace:
I20250113 08:59:28.790432 14927 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:28.775506 11893 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:28.490621 14981 consensus_queue.cc:260] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.638350 14736 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:28.830857 11932 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.864s (SENT) [suppressed 48 similar messages]
I20250113 08:59:28.779991 11967 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:28.780565 12008 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:28.881117 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 100664us
W20250113 08:59:28.781042 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 143453us
W20250113 08:59:28.783402 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 138993us
I20250113 08:59:28.779037 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 773) took 2992 ms. Trace:
W20250113 08:59:28.882342 14971 log.cc:926] Time spent T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 1.081s user 0.002s sys 0.000s
I20250113 08:59:28.792968 11965 raft_consensus.cc:1235] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:28.794013 12041 raft_consensus.cc:1270] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:28.795312 11856 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:28.799943 11888 rpcz_store.cc:276] 0113 08:59:26.989565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.211947 (+222382us) service_pool.cc:225] Handling call
0113 08:59:27.212284 (+ 337us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.437183 (+224899us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.437226 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:28.238581 (+801355us) spinlock_profiling.cc:243] Waited 61.3 ms on lock 0x7b2c00001578. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc334fea741 00007fc334ff9d72 00007fc336645141 00007fc33660958a 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782
0113 08:59:28.239046 (+ 465us) log.cc:844] Serialized 26 byte log entry
0113 08:59:28.482248 (+243202us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:28.482282 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.482353 (+ 71us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.695024 (+212671us) raft_consensus.cc:1680] finished
0113 08:59:28.695051 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:28.697574 (+ 2523us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13937,"raft.queue_time_us":126105,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":83,"spinlock_wait_cycles":140964736,"thread_start_us":433876,"threads_started":2,"wal-append.queue_time_us":404892}
I20250113 08:59:28.804109 12686 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:28.711768 14944 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:28.783941 14881 raft_consensus.cc:2743] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:59:28.819479 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 111733us
I20250113 08:59:28.828878 12724 rpcz_store.cc:276] 0113 08:59:27.656718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.656917 (+ 199us) service_pool.cc:225] Handling call
0113 08:59:27.691318 (+ 34401us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:27.936654 (+245336us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.691617 (+754963us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b540077c254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:28.691788 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":484018432}
I20250113 08:59:28.778571 12039 raft_consensus.cc:2883] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:28.760360 14941 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:28.821163 14960 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.007s sys 0.035s
I20250113 08:59:28.882393 12037 raft_consensus.cc:1270] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250113 08:59:28.761842 14953 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.752s user 0.006s sys 0.023s
I20250113 08:59:28.885362 11859 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:28.885444 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 48003.4us
I20250113 08:59:28.795943 14994 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.840131 14729 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.886772 14736 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:28.887383 14845 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:28.782500 14999 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:28.957480 12042 raft_consensus.cc:1235] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:28.883301 12721 rpcz_store.cc:276] 0113 08:59:25.772987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.133382 (+1360395us) service_pool.cc:225] Handling call
0113 08:59:27.160699 (+ 27317us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.311211 (+150512us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.311241 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.311647 (+ 406us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.520069 (+208422us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:27.520274 (+ 205us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.520374 (+ 100us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.603135 (+1082761us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:28.714195 (+111060us) raft_consensus.cc:1680] finished
0113 08:59:28.714215 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:28.765734 (+ 51519us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58427,"raft.queue_time_us":49926,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":9420,"spinlock_wait_cycles":99310976,"thread_start_us":285023,"threads_started":2,"wal-append.queue_time_us":446525}
I20250113 08:59:28.826073 11884 rpcz_store.cc:276] 0113 08:59:26.952067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.171234 (+219167us) service_pool.cc:225] Handling call
0113 08:59:27.171578 (+ 344us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.553548 (+381970us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.553588 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.573651 (+ 20063us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.707572 (+133921us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:27.837363 (+129791us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.838770 (+ 1407us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.838858 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.664971 (+826113us) spinlock_profiling.cc:243] Waited 45.7 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:28.773784 (+108813us) raft_consensus.cc:1680] finished
0113 08:59:28.773933 (+ 149us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:28.774075 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2396,"prepare.queue_time_us":89826,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":107308,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":21870,"spinlock_wait_cycles":105095040,"thread_start_us":527688,"threads_started":3,"wal-append.queue_time_us":184245}
I20250113 08:59:28.817759 14455 consensus_queue.cc:1046] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20250113 08:59:28.901364 14733 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:28.819240 14976 raft_consensus.cc:738] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:28.884481 11858 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:28.968600 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:28.801914 15000 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.081s
I20250113 08:59:28.840272 14927 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:28.903910 14991 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:28.972594 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 91811.5us
W20250113 08:59:28.939947 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 144750us
I20250113 08:59:28.809880 14911 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:28.948628 12006 leader_election.cc:400] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5.
I20250113 08:59:28.840636 14963 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:28.840282 15001 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:28.947103 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:28.978613 12040 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:28.828495 14647 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
W20250113 08:59:28.951483 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 161659us
I20250113 08:59:28.954998 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:28.929622 14960 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.025s user 0.008s sys 0.036s
W20250113 08:59:28.926764 14953 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.969s user 0.006s sys 0.024s
I20250113 08:59:28.919132 14907 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:28.964979 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:28.970561 14736 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:28.969343 12037 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:28.969556 11968 raft_consensus.cc:1270] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250113 08:59:28.970072 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 62289.5us
I20250113 08:59:28.968909 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1849) took 1910 ms. Trace:
I20250113 08:59:28.972018 14944 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:28.973776 11856 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:28.976218 12006 leader_election.cc:403] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250113 08:59:28.950387 15005 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:28.956295 14795 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:28.926853 14879 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.046s
W20250113 08:59:28.985728 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 103085us
I20250113 08:59:28.964409 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:28.986132 12041 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:28.923750 14766 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:28.997843 11962 raft_consensus.cc:1270] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:28.977772 15000 consensus_queue.cc:1225] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:28.986088 14967 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:29.011320 12038 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:28.991744 11887 rpcz_store.cc:276] 0113 08:59:27.058532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.255445 (+196913us) service_pool.cc:225] Handling call
0113 08:59:27.264198 (+ 8753us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.566148 (+301950us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.566183 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.608782 (+ 42599us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.722873 (+114091us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:27.722942 (+ 69us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.723228 (+ 286us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.702125 (+978897us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:28.955688 (+253563us) raft_consensus.cc:1680] finished
0113 08:59:28.955719 (+ 31us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:28.968843 (+ 13124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19766,"raft.queue_time_us":87812,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"spinlock_wait_cycles":96877440,"thread_start_us":341454,"threads_started":2,"wal-append.queue_time_us":189600}
I20250113 08:59:28.989007 14964 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:28.986457 14960 raft_consensus.cc:695] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:28.988134 14907 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:28.987259 14733 raft_consensus.cc:695] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:28.987196 14953 raft_consensus.cc:695] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:28.987418 14954 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20250113 08:59:28.979561 14963 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:29.301223 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2255) took 4387 ms. Trace:
I20250113 08:59:29.289857 11964 raft_consensus.cc:1270] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250113 08:59:29.316438 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:29.302942 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 639) took 1976 ms (client timeout 1778 ms). Trace:
W20250113 08:59:29.027921 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 54240.6us
I20250113 08:59:29.323648 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:29.145294 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2235) took 2906 ms. Trace:
W20250113 08:59:29.188937 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 358096us
I20250113 08:59:29.329131 12718 raft_consensus.cc:3054] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:29.327860 12717 rpcz_store.cc:269] 0113 08:59:27.326429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.353293 (+ 26864us) service_pool.cc:225] Handling call
0113 08:59:29.302774 (+1949481us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54007026d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:29.302873 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":843677056}
I20250113 08:59:29.012142 14455 raft_consensus.cc:3049] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:29.295902 11858 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:28.994462 14944 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:28.993480 14647 raft_consensus.cc:695] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:29.001335 15001 raft_consensus.cc:695] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:29.000784 14736 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:29.321554 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 310213us
I20250113 08:59:29.354094 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:29.037767 14946 raft_consensus.cc:2758] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250113 08:59:29.352622 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:29.464521 12717 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250113 08:59:29.328258 11963 rpcz_store.cc:276] 0113 08:59:24.914303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.242671 (+2328368us) service_pool.cc:225] Handling call
0113 08:59:27.453969 (+211298us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:27.868359 (+414390us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.868397 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:27.870063 (+ 1666us) log.cc:844] Serialized 52 byte log entry
0113 08:59:28.008970 (+138907us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:28.009021 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.009107 (+ 86us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.728700 (+719593us) raft_consensus.cc:1680] finished
0113 08:59:28.728732 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:29.300951 (+572219us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b540039fb50. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:29.301159 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":89603,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":13672,"spinlock_wait_cycles":387669760,"thread_start_us":186551,"threads_started":2,"wal-append.queue_time_us":181563}
I20250113 08:59:29.824585 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2280) took 1090 ms. Trace:
I20250113 08:59:29.834311 11968 rpcz_store.cc:276] 0113 08:59:28.734270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.744264 (+ 9994us) service_pool.cc:225] Handling call
0113 08:59:28.776121 (+ 31857us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:28.990991 (+214870us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:29.644050 (+653059us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540006d6d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:29.824500 (+180450us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":444064128}
I20250113 08:59:29.346290 12721 raft_consensus.cc:1235] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:29.352221 12719 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250113 08:59:29.353128 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:29.832878 12722 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:29.584230 12718 raft_consensus.cc:2463] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:29.832232 11859 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
W20250113 08:59:29.827615 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.03347e+06us
I20250113 08:59:29.631242 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2261) took 3235 ms. Trace:
I20250113 08:59:29.838094 12686 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:29.838042 11965 rpcz_store.cc:276] 0113 08:59:26.308248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.321407 (+1013159us) service_pool.cc:225] Handling call
0113 08:59:27.321735 (+ 328us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:28.905711 (+1583976us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:29.542843 (+637132us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400050fd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:29.543118 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3383092992}
W20250113 08:59:29.830231 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 905194us
I20250113 08:59:29.329741 11966 rpcz_store.cc:276] 0113 08:59:26.241198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.311175 (+1069977us) service_pool.cc:225] Handling call
0113 08:59:27.311612 (+ 437us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:27.359191 (+ 47579us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:27.359237 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:27.417850 (+ 58613us) log.cc:844] Serialized 26 byte log entry
0113 08:59:27.693399 (+275549us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:27.723199 (+ 29800us) log.cc:844] Serialized 14 byte log entry
0113 08:59:27.951446 (+228247us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:27.960747 (+ 9301us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:28.881563 (+920816us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:29.038117 (+156554us) raft_consensus.cc:1680] finished
0113 08:59:29.145011 (+106894us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:29.145185 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32860,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":128436,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":31215,"spinlock_wait_cycles":93966080,"thread_start_us":410857,"threads_started":2,"wal-append.queue_time_us":230784}
I20250113 08:59:29.341557 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:29.840006 11961 raft_consensus.cc:2405] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250113 08:59:29.302385 14940 consensus_meta.cc:287] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.561s user 0.010s sys 0.001s
W20250113 08:59:29.739846 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.002s sys 0.010s
W20250113 08:59:29.028916 15012 log.cc:926] Time spent T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
I20250113 08:59:29.337630 14994 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:29.876549 14994 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:29.289510 15027 consensus_queue.cc:1035] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.613s
W20250113 08:59:29.155587 15018 log.cc:926] Time spent T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
W20250113 08:59:29.870836 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.880s user 0.004s sys 0.016s
W20250113 08:59:29.877593 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 886626us
I20250113 08:59:29.836200 14954 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:29.872560 15005 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election won for term 5
W20250113 08:59:29.343307 14723 consensus_meta.cc:287] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.567s user 0.007s sys 0.002s
I20250113 08:59:29.473340 14992 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:29.156002 15007 consensus_queue.cc:1035] T 4a9d15ff319f43f0a1dd5ae717393675 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.718s
W20250113 08:59:29.837064 14993 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.012s sys 0.005s
I20250113 08:59:29.409211 14967 raft_consensus.cc:695] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:29.873594 14601 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:29.302469 14927 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:29.407442 14907 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:29.342528 14664 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:29.303125 14865 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:29.343358 14983 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:29.425637 15031 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:29.781718 14798 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.005s sys 0.015s
I20250113 08:59:29.835525 14964 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:29.156930 14406 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:29.339704 14790 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:29.848167 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 552326us
I20250113 08:59:29.299271 14999 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:29.035337 15021 consensus_queue.cc:1046] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.879s
I20250113 08:59:29.028323 15015 consensus_queue.cc:1046] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.845s
W20250113 08:59:29.652437 15028 log.cc:926] Time spent T 00000000000000000000000000000000 P 48f8007a91af4c0c837ee737c6326d99: Append to log took a long time: real 0.273s user 0.000s sys 0.002s
I20250113 08:59:29.036605 14911 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:29.032814 15020 log.cc:926] Time spent T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20250113 08:59:29.790503 14991 consensus_meta.cc:287] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.813s user 0.009s sys 0.001s
I20250113 08:59:29.892411 14994 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:29.894841 12036 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:29.028429 14970 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 4. Current term is 5.
I20250113 08:59:29.835180 14944 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:29.897265 11884 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20250113 08:59:29.896054 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:29.902107 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:29.842950 15019 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:29.787501 14963 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:29.904841 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 503) took 2904 ms (client timeout 1752 ms). Trace:
I20250113 08:59:29.904927 14941 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:29.035543 15024 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.885s
I20250113 08:59:29.905586 11959 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:29.155871 14729 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:29.703493 14960 consensus_queue.cc:237] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:29.426241 14733 consensus_queue.cc:237] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:29.906526 15027 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:29.842615 14881 consensus_queue.cc:1035] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20250113 08:59:29.878309 14954 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:29.907836 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 789) took 1608 ms. Trace:
I20250113 08:59:29.830271 14455 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:29.020390 14766 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:29.917622 14601 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:29.353438 14951 consensus_queue.cc:1046] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.545s
I20250113 08:59:29.337380 14950 consensus_queue.cc:1046] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.566s
I20250113 08:59:29.020061 14976 consensus_queue.cc:260] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:29.704674 14953 consensus_queue.cc:237] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:29.119282 14795 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:29.357434 15030 consensus_queue.cc:1035] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.706s
W20250113 08:59:30.476933 12724 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.009s sys 0.011s
W20250113 08:59:30.644963 12724 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.014s sys 0.013s
I20250113 08:59:30.645823 14455 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.646548 14983 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:30.647430 14455 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:30.602643 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 764566us
W20250113 08:59:29.946756 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 75267.9us
I20250113 08:59:30.649706 12686 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:30.650386 12006 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:30.652096 14601 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:30.652758 14601 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:29.898742 14940 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:59:30.640472 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.002s sys 0.006s
W20250113 08:59:30.605880 11960 rpcz_store.cc:269] 0113 08:59:27.000802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.061717 (+ 60915us) service_pool.cc:225] Handling call
0113 08:59:29.904621 (+2842904us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b540006e0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:29.904767 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1566025984}
W20250113 08:59:30.393909 12007 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.002s sys 0.019s
W20250113 08:59:30.025170 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 194447us
I20250113 08:59:30.079473 14927 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:30.654794 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.002s sys 0.006s
I20250113 08:59:30.605856 11887 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:30.477089 12720 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.002s sys 0.019s
W20250113 08:59:30.662413 12720 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.801s user 0.003s sys 0.024s
I20250113 08:59:30.663364 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 800) took 1314 ms. Trace:
W20250113 08:59:30.472939 11859 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.012s sys 0.008s
I20250113 08:59:30.656827 11856 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
W20250113 08:59:30.665290 11859 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.792s user 0.017s sys 0.009s
W20250113 08:59:30.655762 12007 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.783s user 0.005s sys 0.021s
W20250113 08:59:30.666462 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 843124us
W20250113 08:59:30.669409 12006 connection.cc:232] RPC call timeout handler was delayed by -3.5585908809999864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
W20250113 08:59:29.902117 14993 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.603s user 0.013s sys 0.006s
I20250113 08:59:30.674396 12033 raft_consensus.cc:1270] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250113 08:59:30.674847 12033 pending_rounds.cc:77] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 7
I20250113 08:59:30.675110 12033 pending_rounds.cc:99] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250113 08:59:30.675380 12033 raft_consensus.cc:2883] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:30.663905 12720 rpcz_store.cc:276] 0113 08:59:29.348951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:29.414749 (+ 65798us) service_pool.cc:225] Handling call
0113 08:59:29.415094 (+ 345us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:29.415322 (+ 228us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:29.473802 (+ 58480us) log.cc:844] Serialized 14 byte log entry
0113 08:59:30.662932 (+1189130us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:30.662997 (+ 65us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:30.663061 (+ 64us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:30.663150 (+ 89us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:30.663296 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":361311,"prepare.queue_time_us":100856,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":68,"raft.queue_time_us":432258,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":801483,"threads_started":1}
W20250113 08:59:30.676271 12006 connection.cc:232] RPC call timeout handler was delayed by -3.5414908809999588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
W20250113 08:59:30.676934 12006 connection.cc:232] RPC call timeout handler was delayed by -3.3218908809999448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
W20250113 08:59:30.677269 12006 connection.cc:232] RPC call timeout handler was delayed by -3.2606908809999595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20250113 08:59:30.677593 12006 connection.cc:232] RPC call timeout handler was delayed by -3.2588908809999566s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
W20250113 08:59:30.677855 12006 connection.cc:232] RPC call timeout handler was delayed by -3.0014908809999952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20250113 08:59:30.649755 14994 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:30.389483 14946 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
W20250113 08:59:30.678333 12006 connection.cc:232] RPC call timeout handler was delayed by -2.986096987999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
I20250113 08:59:30.656554 14966 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:30.644318 14798 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.646s user 0.010s sys 0.036s
W20250113 08:59:30.677779 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.004s sys 0.009s
I20250113 08:59:30.189874 15040 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.670001 14865 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.672466 14911 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.247424 15005 raft_consensus.cc:695] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:30.653218 14601 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:30.730322 14601 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:30.730625 15005 consensus_queue.cc:237] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:30.689415 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.524s user 0.004s sys 0.009s
I20250113 08:59:30.476668 14717 consensus_queue.cc:1046] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.207s
I20250113 08:59:29.888496 14723 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:30.686606 14729 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.692358 14966 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.693114 14954 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:30.693707 15039 log.cc:926] Time spent T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.226s user 0.000s sys 0.001s
I20250113 08:59:30.077481 14736 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:30.672750 14881 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.658453 14964 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:30.706060 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 873843us
I20250113 08:59:30.782352 11859 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:30.605787 12726 rpcz_store.cc:276] 0113 08:59:28.299472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.299672 (+ 200us) service_pool.cc:225] Handling call
0113 08:59:28.332702 (+ 33030us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:28.587399 (+254697us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:28.587445 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:28.594319 (+ 6874us) log.cc:844] Serialized 52 byte log entry
0113 08:59:28.786128 (+191809us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:28.819071 (+ 32943us) log.cc:844] Serialized 14 byte log entry
0113 08:59:28.819296 (+ 225us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:28.819374 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:29.907631 (+1088257us) raft_consensus.cc:1680] finished
0113 08:59:29.907650 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:29.907776 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":261918,"prepare.queue_time_us":38383,"prepare.run_cpu_time_us":9047,"prepare.run_wall_time_us":42983,"raft.queue_time_us":22928,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":6403,"spinlock_wait_cycles":9615104,"thread_start_us":181997,"threads_started":1,"wal-append.queue_time_us":368378}
W20250113 08:59:30.737465 12006 connection.cc:232] RPC call timeout handler was delayed by -2.8241908809999359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.244s for a response
I20250113 08:59:30.687489 15033 consensus_queue.cc:1035] T 6a1ace8d6d3d488583610962dabdee94 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
W20250113 08:59:30.743124 12005 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.000s sys 0.038s
W20250113 08:59:30.783859 12006 connection.cc:232] RPC call timeout handler was delayed by -2.7728908809999666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2497s for a response
W20250113 08:59:30.784101 12005 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.910s user 0.000s sys 0.038s
I20250113 08:59:30.743645 14951 raft_consensus.cc:3049] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
W20250113 08:59:30.755158 11932 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.023s sys 0.035s
I20250113 08:59:30.755003 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:30.653851 14940 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:30.744096 14946 raft_consensus.cc:695] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:30.758174 12686 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:30.814554 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 56440.8us
I20250113 08:59:30.658313 14790 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.725237 14927 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.763223 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:30.766242 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 121586us
I20250113 08:59:30.767457 15021 consensus_queue.cc:1225] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:29.898016 14992 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.744865 14827 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.079053 14647 consensus_queue.cc:237] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:30.649112 14983 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:30.768687 12687 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.011s sys 0.025s
I20250113 08:59:30.777580 14795 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:30.777740 15056 consensus_queue.cc:1046] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:30.778064 15045 consensus_queue.cc:1046] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250113 08:59:30.642539 15019 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:30.778496 14723 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:30.778671 14954 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250113 08:59:30.779477 14881 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.397390 15001 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.734616 15051 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:30.070638 14731 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:30.807734 11932 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.954s user 0.024s sys 0.035s
I20250113 08:59:30.893283 15021 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:30.924679 11933 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.005s sys 0.008s
W20250113 08:59:30.935176 11933 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.098s user 0.005s sys 0.009s
I20250113 08:59:30.560227 14967 consensus_queue.cc:237] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.816452 11931 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:30.946597 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 130221us
I20250113 08:59:30.947539 11931 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:30.818514 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:30.821569 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:30.822500 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:30.830013 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 173254us
I20250113 08:59:30.744796 14911 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.833115 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:30.955555 12038 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 6
I20250113 08:59:30.834054 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:30.723713 15040 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:30.865226 12687 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.035s user 0.012s sys 0.025s
I20250113 08:59:30.796558 14951 raft_consensus.cc:738] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:30.674091 15031 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.765185 14729 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:30.903313 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 121064us
I20250113 08:59:30.813217 15027 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:30.924721 15051 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:30.831436 14970 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.925448 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1729) took 1979 ms. Trace:
I20250113 08:59:30.949589 15021 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:30.860294 14941 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:59:30.818187 12006 connection.cc:232] RPC call timeout handler was delayed by -2.5757062599999472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
W20250113 08:59:31.000908 12006 connection.cc:232] RPC call timeout handler was delayed by -2.1149760339999375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18130000000000002s for a response
W20250113 08:59:31.001240 12006 connection.cc:232] RPC call timeout handler was delayed by -2.0691353609998941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
W20250113 08:59:31.001549 12006 connection.cc:232] RPC call timeout handler was delayed by -1.9488226359999317s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2609s for a response
W20250113 08:59:31.001824 12006 connection.cc:232] RPC call timeout handler was delayed by -1.8054353609999225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20250113 08:59:31.002094 12006 connection.cc:232] RPC call timeout handler was delayed by -1.7757353609998745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1988s for a response
I20250113 08:59:31.002306 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
W20250113 08:59:30.933760 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 118290us
I20250113 08:59:31.003108 11968 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:31.002343 12006 connection.cc:232] RPC call timeout handler was delayed by -1.7613353609999649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
W20250113 08:59:31.003799 12006 connection.cc:232] RPC call timeout handler was delayed by -1.6596572289998903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29209999999999997s for a response
W20250113 08:59:31.004066 12006 connection.cc:232] RPC call timeout handler was delayed by -1.0610908809999273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4399s for a response
I20250113 08:59:30.962693 11931 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:30.970275 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:30.921478 15015 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:30.861012 14927 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:30.981050 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:30.981844 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:31.006722 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:30.925750 15056 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:31.015897 14729 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.016310 15056 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.017953 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:30.831516 14991 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:30.892635 14827 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.027433 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:30.997920 12033 rpcz_store.cc:276] 0113 08:59:28.934059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.944980 (+ 10921us) service_pool.cc:225] Handling call
0113 08:59:28.945388 (+ 408us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:30.675865 (+1730477us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:30.913313 (+237448us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54002b8c54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:30.913503 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3975569792}
I20250113 08:59:31.028877 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:30.831432 14999 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.999156 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:30.949456 14983 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:31.005797 14766 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:31.007467 11891 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:31.007725 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:31.008387 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 45770.2us
I20250113 08:59:31.008447 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.056484 11931 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:31.009776 15048 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:31.010011 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:31.010188 15053 raft_consensus.cc:3049] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:31.010656 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:31.010285 14951 consensus_queue.cc:260] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:31.012707 14911 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.014138 14927 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:31.015264 15031 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.019079 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:31.005913 14717 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:31.092681 14717 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:31.069468 14907 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:31.093362 14717 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:31.095232 14907 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:31.019034 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 46512.1us
I20250113 08:59:31.096787 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 597) took 2320 ms. Trace:
I20250113 08:59:31.021526 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:31.097169 12042 rpcz_store.cc:276] 0113 08:59:28.776053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.781933 (+ 5880us) service_pool.cc:225] Handling call
0113 08:59:28.803251 (+ 21318us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:28.971160 (+167909us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.096548 (+2125388us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b540021d754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:31.096730 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":352825088}
I20250113 08:59:31.099449 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:31.030145 11934 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:31.030711 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:31.031245 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 51947us
I20250113 08:59:31.032825 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:31.033481 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:31.036345 12038 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:30.949380 15045 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.059s
I20250113 08:59:31.104461 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.105419 12038 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:30.996784 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:31.054656 11963 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:31.008436 15021 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:31.054898 14978 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:31.116482 11859 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:31.116822 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:31.117826 11968 raft_consensus.cc:2388] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:31.119282 12722 raft_consensus.cc:2405] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250113 08:59:31.066185 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:31.059228 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.059345 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:31.120574 11888 raft_consensus.cc:2388] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:31.063946 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:31.067716 12035 raft_consensus.cc:2388] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 8: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:31.000236 15051 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:31.069918 12721 raft_consensus.cc:1270] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:31.063755 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:31.066160 14963 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:31.069469 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:31.078699 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:31.027973 14729 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:30.996819 14723 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.068s
I20250113 08:59:31.053988 14954 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:31.032373 15056 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:31.126127 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:31.065430 14865 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.136235 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:31.130437 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:31.071749 15053 raft_consensus.cc:738] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:31.130086 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.008064 14970 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.142844 11859 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:31.053840 14991 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:31.150354 12038 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250113 08:59:31.166148 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 136093us
I20250113 08:59:31.168768 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.180073 12042 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:31.168771 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.058486 14999 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.064011 14941 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.173390 11931 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:31.176620 12684 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:31.000116 15027 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.058810 14983 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.126485 14963 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:31.126335 14911 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:31.126938 15040 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:31.190785 11934 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:31.032639 14946 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.195899 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:31.197168 12035 raft_consensus.cc:2405] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250113 08:59:31.199349 11887 consensus_meta.cc:287] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.522s user 0.009s sys 0.000s
I20250113 08:59:31.215999 11887 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:31.200513 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 57796.8us
I20250113 08:59:31.202057 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:31.180231 15055 log.cc:926] Time spent T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20250113 08:59:31.166818 14940 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:31.178364 14865 raft_consensus.cc:2798] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:31.213583 11934 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:31.200495 15058 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:31.138881 14954 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.216646 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 608) took 1337 ms. Trace:
I20250113 08:59:31.128003 14978 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.217962 12041 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:31.205372 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 160686us
I20250113 08:59:31.283370 11859 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:31.285012 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:31.373515 12040 raft_consensus.cc:2388] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:31.217156 15040 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:31.196573 14790 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.201195 14963 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:31.134454 14729 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.070338 15048 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:31.241542 11887 rpcz_store.cc:276] 0113 08:59:29.879287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:29.879431 (+ 144us) service_pool.cc:225] Handling call
0113 08:59:31.216578 (+1337147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:31.364506 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.218209 14881 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.216215 14992 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.371045 12718 raft_consensus.cc:2371] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6. Re-sending same reply.
I20250113 08:59:31.368101 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:31.224752 15056 consensus_queue.cc:1035] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.150s
I20250113 08:59:31.181802 14964 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:31.201022 14736 consensus_queue.cc:1046] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.167s
I20250113 08:59:31.383286 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:31.402797 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:31.404950 11887 pending_rounds.cc:77] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 4
I20250113 08:59:31.406250 12038 raft_consensus.cc:2388] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:31.478057 12040 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Advancing to term 11
I20250113 08:59:31.478570 11887 pending_rounds.cc:99] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.5
W20250113 08:59:31.476497 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 74708.6us
I20250113 08:59:31.477979 12042 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:31.497606 12034 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.003s sys 0.006s
I20250113 08:59:31.490587 12686 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:31.204356 15031 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:31.215509 14911 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.370993 14723 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:31.202809 15034 consensus_queue.cc:1035] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.420s
I20250113 08:59:31.490002 11887 raft_consensus.cc:2883] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:31.393986 15080 log.cc:926] Time spent T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.203s user 0.000s sys 0.000s
W20250113 08:59:31.393206 15074 log.cc:926] Time spent T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.311s user 0.001s sys 0.000s
W20250113 08:59:31.499212 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 320788us
W20250113 08:59:31.498206 12034 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.003s sys 0.006s
W20250113 08:59:31.396271 15063 log.cc:926] Time spent T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
W20250113 08:59:31.498837 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 400880us
I20250113 08:59:31.517445 11931 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:31.398775 14978 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:31.390789 15070 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:31.394979 15052 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 08:59:31.539099 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:31.613583 12038 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:31.478199 15019 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:31.394831 15059 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:31.391768 15073 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:31.549283 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:31.392319 15065 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:31.523411 15045 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:31.391047 15061 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250113 08:59:31.682971 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 72214.4us
I20250113 08:59:31.394685 15068 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:31.365083 15060 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.479s
I20250113 08:59:31.412833 14991 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:31.611153 12042 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:31.682615 14795 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:31.511049 15031 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:31.666294 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:31.682627 15061 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:31.674364 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:31.706517 14621 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20250113 08:59:31.297187 15058 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:31.405944 14790 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.201s
W20250113 08:59:31.706403 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 95805.5us
W20250113 08:59:31.704511 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 480054us
I20250113 08:59:31.606590 14927 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:31.702669 15048 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:31.395133 15067 consensus_queue.cc:1046] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.605s
I20250113 08:59:31.664991 15059 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:31.827922 11859 leader_election.cc:400] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 8. Current term is 9.
W20250113 08:59:31.728144 14990 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.005s sys 0.003s
I20250113 08:59:31.729945 15073 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:31.724259 14970 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:31.704638 14717 consensus_meta.cc:287] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.611s user 0.005s sys 0.002s
I20250113 08:59:31.734309 14911 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.734438 15049 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:31.725231 15045 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:31.394511 15071 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.653s
I20250113 08:59:31.736047 14954 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:31.380131 15053 consensus_queue.cc:260] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:31.737881 15087 log.cc:926] Time spent T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250113 08:59:31.833966 15068 raft_consensus.cc:695] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:31.744999 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:31.501658 14827 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.769018 14766 consensus_queue.cc:1035] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.096s
I20250113 08:59:31.756742 12038 raft_consensus.cc:2463] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
W20250113 08:59:31.705663 14601 consensus_meta.cc:287] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.975s user 0.011s sys 0.006s
I20250113 08:59:31.828455 11859 leader_election.cc:403] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:31.830233 12040 raft_consensus.cc:2405] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20250113 08:59:31.770447 15031 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:31.834911 14990 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.911s user 0.005s sys 0.004s
I20250113 08:59:31.801374 15061 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:31.810667 14941 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.810437 14999 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:31.526058 14723 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.813424 14983 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:31.673084 12037 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.006s sys 0.016s
I20250113 08:59:31.800562 14795 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:31.674522 15070 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:31.881026 14729 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:32.150920 12723 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5"
dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
mode: GRACEFUL
new_leader_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:56200
W20250113 08:59:32.016239 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.788s user 0.007s sys 0.008s
I20250113 08:59:32.151702 12723 raft_consensus.cc:604] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Received request to transfer leadership to TS 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:32.152056 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.924s user 0.011s sys 0.011s
W20250113 08:59:32.152395 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 924527us
I20250113 08:59:32.153646 11934 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:32.154739 14723 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:32.156060 14601 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:32.015916 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 473893us
I20250113 08:59:32.157385 12033 raft_consensus.cc:1235] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:31.854784 15073 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:32.158470 10702 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250113 08:59:31.864627 14717 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:32.160110 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:32.160524 15097 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.457s user 0.000s sys 0.001s
I20250113 08:59:32.160837 11965 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:32.161120 12042 raft_consensus.cc:2405] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20250113 08:59:32.163743 12037 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.099s user 0.015s sys 0.016s
W20250113 08:59:32.156430 12006 leader_election.cc:336] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.887s (SENT)
I20250113 08:59:32.165681 12006 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:32.158396 14601 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:31.834434 15059 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:32.161221 10702 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250113 08:59:32.144297 14944 consensus_meta.cc:287] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.473s user 0.008s sys 0.005s
I20250113 08:59:32.340660 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2289) took 1369 ms. Trace:
I20250113 08:59:32.177327 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:32.340345 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 798972us
I20250113 08:59:32.162052 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:32.553750 14954 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:32.817620 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1747) took 1685 ms. Trace:
I20250113 08:59:32.554240 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:32.380760 15079 log.cc:926] Time spent T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.997s user 0.000s sys 0.003s
I20250113 08:59:32.818773 12720 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
W20250113 08:59:32.818990 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 662585us
I20250113 08:59:32.358526 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1735) took 3421 ms. Trace:
W20250113 08:59:32.107787 14455 consensus_meta.cc:287] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.460s user 0.007s sys 0.007s
W20250113 08:59:32.542080 11934 leader_election.cc:336] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.778s (SENT)
I20250113 08:59:32.820794 11934 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:31.880980 15045 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:32.368497 14963 consensus_meta.cc:287] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.960s user 0.008s sys 0.004s
I20250113 08:59:32.661985 15000 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:31.960691 15027 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:32.233971 15058 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:32.342969 14993 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.009s sys 0.027s
I20250113 08:59:32.553607 11856 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:32.823437 14993 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.083s user 0.009s sys 0.041s
I20250113 08:59:32.303872 14983 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:32.824272 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 270774us
I20250113 08:59:32.756156 15065 raft_consensus.cc:2743] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511) with higher term. Message: Invalid argument: T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 8. Current term is 9.
I20250113 08:59:32.824110 15015 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:32.229043 14729 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:32.339659 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 601) took 1553 ms. Trace:
W20250113 08:59:32.230105 15081 log.cc:926] Time spent T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.185s user 0.000s sys 0.000s
I20250113 08:59:32.818859 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:32.826864 12725 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:32.819798 14954 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:32.234028 15103 log.cc:926] Time spent T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.222s user 0.001s sys 0.000s
I20250113 08:59:32.178804 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2291) took 1004 ms. Trace:
I20250113 08:59:32.358875 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:32.551327 15021 consensus_meta.cc:287] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.406s user 0.012s sys 0.001s
I20250113 08:59:32.341434 11968 rpcz_store.cc:276] 0113 08:59:30.971146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.118409 (+147263us) service_pool.cc:225] Handling call
0113 08:59:31.118757 (+ 348us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:31.136187 (+ 17430us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:31.136218 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:31.148116 (+ 11898us) log.cc:844] Serialized 26 byte log entry
0113 08:59:31.365293 (+217177us) raft_consensus.cc:1640] Marking committed up to 1
0113 08:59:31.365588 (+ 295us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.367124 (+ 1536us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:32.340283 (+973159us) raft_consensus.cc:1680] finished
0113 08:59:32.340305 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:32.340601 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7456,"raft.queue_time_us":23174,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":22189,"thread_start_us":149720,"threads_started":1,"wal-append.queue_time_us":974533}
I20250113 08:59:32.365832 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2281) took 3630 ms. Trace:
I20250113 08:59:32.831424 11967 rpcz_store.cc:276] 0113 08:59:28.735363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.736299 (+ 936us) service_pool.cc:225] Handling call
0113 08:59:28.736715 (+ 416us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:28.890031 (+153316us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.365489 (+3475458us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b5400196754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:32.365752 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89581312}
I20250113 08:59:32.239233 15071 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:32.827333 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 611) took 2089 ms. Trace:
I20250113 08:59:32.827427 12726 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
W20250113 08:59:32.234143 15097 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250113 08:59:32.822194 15000 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:32.899294 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:32.454258 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:32.820003 12036 rpcz_store.cc:276] 0113 08:59:28.937272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.975620 (+ 38348us) service_pool.cc:225] Handling call
0113 08:59:28.989385 (+ 13765us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:30.649322 (+1659937us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.358294 (+1708972us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54003bc754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:32.358452 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2081476352}
I20250113 08:59:32.468892 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:32.905664 12722 raft_consensus.cc:2388] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 8: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:32.818207 12042 rpcz_store.cc:276] 0113 08:59:31.132327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.161792 (+1029465us) service_pool.cc:225] Handling call
0113 08:59:32.817557 (+655765us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:32.839797 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 297755us
I20250113 08:59:32.824643 14983 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:32.822052 14963 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:32.826126 12034 rpcz_store.cc:276] 0113 08:59:30.786043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:30.787239 (+ 1196us) service_pool.cc:225] Handling call
0113 08:59:30.832442 (+ 45203us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:30.905773 (+ 73331us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:30.905826 (+ 53us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:30.906497 (+ 671us) log.cc:844] Serialized 52 byte log entry
0113 08:59:31.540981 (+634484us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:31.714357 (+173376us) log.cc:844] Serialized 14 byte log entry
0113 08:59:31.876976 (+162619us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.877093 (+ 117us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:32.339320 (+462227us) raft_consensus.cc:1680] finished
0113 08:59:32.339349 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:32.339573 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29289,"prepare.queue_time_us":133941,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":85576,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":22988,"thread_start_us":700769,"threads_started":2,"wal-append.queue_time_us":1084723}
I20250113 08:59:32.828483 12040 rpcz_store.cc:276] 0113 08:59:31.174480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.383025 (+208545us) service_pool.cc:225] Handling call
0113 08:59:32.178728 (+795703us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:32.368806 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2288) took 1398 ms. Trace:
I20250113 08:59:32.921586 11964 rpcz_store.cc:276] 0113 08:59:30.970431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:30.970604 (+ 173us) service_pool.cc:225] Handling call
0113 08:59:32.367818 (+1397214us) spinlock_profiling.cc:243] Waited 936 us on lock 0x7b58001c4198. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:59:32.368099 (+ 281us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.368364 (+ 265us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:32.368412 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:32.368469 (+ 57us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:32.368529 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.368602 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:32.368732 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2153728}
I20250113 08:59:32.825181 11856 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:32.924189 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 99089.3us
I20250113 08:59:32.924924 12042 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:32.897691 11887 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.059s user 0.000s sys 0.033s
I20250113 08:59:32.925565 11856 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:32.926219 11887 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.088s user 0.000s sys 0.034s
I20250113 08:59:32.928035 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:32.928418 12040 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
W20250113 08:59:32.892357 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 1.06453e+06us
I20250113 08:59:32.928901 12038 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 4.
I20250113 08:59:32.175498 15073 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:32.883873 11889 rpcz_store.cc:276] 0113 08:59:30.737568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:30.737741 (+ 173us) service_pool.cc:225] Handling call
0113 08:59:30.768753 (+ 31012us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:30.831747 (+ 62994us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:30.831780 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:30.913059 (+ 81279us) log.cc:844] Serialized 52 byte log entry
0113 08:59:31.025708 (+112649us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:31.025767 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.025862 (+ 95us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:32.827121 (+1801259us) raft_consensus.cc:1680] finished
0113 08:59:32.827140 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:32.827275 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52711,"raft.queue_time_us":120417,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":15318,"thread_start_us":37219,"threads_started":1,"wal-append.queue_time_us":820228}
W20250113 08:59:32.232254 15051 consensus_meta.cc:287] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.096s user 0.006s sys 0.004s
I20250113 08:59:32.930989 11856 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:32.919948 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:32.933661 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1744) took 1813 ms. Trace:
I20250113 08:59:32.927518 15106 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:32.939288 12042 rpcz_store.cc:276] 0113 08:59:31.120530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.924339 (+1803809us) service_pool.cc:225] Handling call
0113 08:59:32.924726 (+ 387us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.933340 (+ 8614us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.933590 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:32.941005 15015 raft_consensus.cc:2743] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:32.941450 12042 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250113 08:59:32.942235 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2299) took 1701 ms. Trace:
I20250113 08:59:32.942358 12034 raft_consensus.cc:1235] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:32.943167 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2290) took 1769 ms. Trace:
I20250113 08:59:32.943573 12034 rpcz_store.cc:276] 0113 08:59:31.174160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.929104 (+1754944us) service_pool.cc:225] Handling call
0113 08:59:32.942136 (+ 13032us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.942878 (+ 742us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.943107 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:32.945363 15093 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:32.946316 15093 consensus_queue.cc:1035] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250113 08:59:32.948060 15093 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:32.949579 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:32.820791 14455 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:32.288291 14827 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:32.934906 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2294) took 1759 ms. Trace:
I20250113 08:59:32.963073 12040 rpcz_store.cc:276] 0113 08:59:31.175359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.927774 (+1752415us) service_pool.cc:225] Handling call
0113 08:59:32.928183 (+ 409us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.934560 (+ 6377us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.934830 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:32.885131 14795 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:32.902925 15058 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:32.344071 15054 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:32.817094 15053 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:32.951367 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:32.901049 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:32.942660 12042 rpcz_store.cc:276] 0113 08:59:31.240863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.940571 (+1699708us) service_pool.cc:225] Handling call
0113 08:59:32.940986 (+ 415us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.941944 (+ 958us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.942170 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:32.956177 11856 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:32.956375 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:32.987211 12725 raft_consensus.cc:2405] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:32.814460 15070 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:32.898689 15108 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:32.984356 11968 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:32.999527 12006 leader_election.cc:336] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.243s (SENT)
I20250113 08:59:33.000195 12006 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:32.983026 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1989) took 1995 ms. Trace:
I20250113 08:59:32.958341 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:32.971305 14731 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.016s sys 0.035s
W20250113 08:59:32.962159 12037 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.014s sys 0.019s
W20250113 08:59:33.013154 12037 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.849s user 0.014s sys 0.019s
I20250113 08:59:32.342851 14990 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:32.967343 12038 raft_consensus.cc:1270] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:59:32.973745 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:32.991027 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:32.968137 15015 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:33.019889 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1739) took 2207 ms. Trace:
I20250113 08:59:33.035722 12037 rpcz_store.cc:276] 0113 08:59:30.812885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:30.972835 (+159950us) service_pool.cc:225] Handling call
0113 08:59:31.006770 (+ 33935us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:31.007001 (+ 231us) raft_consensus.cc:1529] Early marking committed up to index 3
0113 08:59:32.164320 (+1157319us) log.cc:844] Serialized 14 byte log entry
0113 08:59:33.019464 (+855144us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:33.019531 (+ 67us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:33.019585 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.019672 (+ 87us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:33.019819 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5663,"prepare.queue_time_us":384248,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":29,"thread_start_us":1949007,"threads_started":2}
I20250113 08:59:33.022089 12036 raft_consensus.cc:2463] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:33.030159 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2298) took 1809 ms. Trace:
I20250113 08:59:33.038561 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1746) took 1910 ms. Trace:
I20250113 08:59:33.030369 11964 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:33.038991 12036 rpcz_store.cc:276] 0113 08:59:31.128387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.925529 (+1797142us) service_pool.cc:225] Handling call
0113 08:59:33.038504 (+112975us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:33.001416 11890 rpcz_store.cc:276] 0113 08:59:30.987586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.074743 (+ 87157us) service_pool.cc:225] Handling call
0113 08:59:31.097758 (+ 23015us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:31.123764 (+ 26006us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:31.123805 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:31.146372 (+ 22567us) log.cc:844] Serialized 26 byte log entry
0113 08:59:31.511300 (+364928us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:31.655727 (+144427us) log.cc:844] Serialized 14 byte log entry
0113 08:59:31.656001 (+ 274us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.656107 (+ 106us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:32.159742 (+503635us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:32.963656 (+803914us) raft_consensus.cc:1680] finished
0113 08:59:32.982704 (+ 19048us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:32.982941 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33037,"prepare.queue_time_us":167369,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":80,"raft.queue_time_us":137716,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":10284,"spinlock_wait_cycles":43544320,"thread_start_us":273526,"threads_started":1,"wal-append.queue_time_us":465917}
I20250113 08:59:32.882658 14944 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.032692 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.040134 11968 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:33.041039 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.038515 12038 rpcz_store.cc:276] 0113 08:59:31.220632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.929737 (+1709105us) service_pool.cc:225] Handling call
0113 08:59:32.966887 (+ 37150us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:33.029860 (+ 62973us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.030092 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:33.011494 14927 consensus_meta.cc:287] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.178s user 0.013s sys 0.000s
W20250113 08:59:33.013051 14731 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s user 0.017s sys 0.035s
I20250113 08:59:33.049402 12038 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20250113 08:59:32.961423 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2300) took 1589 ms. Trace:
I20250113 08:59:33.049667 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.052170 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2258) took 1685 ms. Trace:
I20250113 08:59:33.056435 11891 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:33.061574 12725 raft_consensus.cc:1235] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:33.061622 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:31.872682 15049 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.063027 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:33.080891 11889 raft_consensus.cc:2388] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:32.902575 15021 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:32.940168 15106 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:32.958937 15051 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:33.065541 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 99558.6us
I20250113 08:59:33.066324 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:32.973114 14455 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.066111 12034 rpcz_store.cc:276] 0113 08:59:31.372349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.944510 (+1572161us) service_pool.cc:225] Handling call
0113 08:59:32.944865 (+ 355us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.945078 (+ 213us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:32.945114 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:32.945169 (+ 55us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:32.945220 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.961184 (+ 15964us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:32.961356 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:33.070142 12720 raft_consensus.cc:1235] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:33.070958 11964 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:33.069764 11963 rpcz_store.cc:276] 0113 08:59:31.366593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.366743 (+ 150us) service_pool.cc:225] Handling call
0113 08:59:31.382928 (+ 16185us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:31.383272 (+ 344us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:31.383301 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:31.403672 (+ 20371us) log.cc:844] Serialized 52 byte log entry
0113 08:59:31.677899 (+274227us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:31.677948 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.678026 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:33.051882 (+1373856us) raft_consensus.cc:1680] finished
0113 08:59:33.051912 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:33.052094 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":142538,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":14103,"thread_start_us":176990,"threads_started":1,"wal-append.queue_time_us":403298}
W20250113 08:59:33.047420 15117 log.cc:926] Time spent T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.276s user 0.001s sys 0.000s
I20250113 08:59:33.072726 14944 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.074016 12038 raft_consensus.cc:1270] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250113 08:59:32.909740 15071 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.079612 11890 raft_consensus.cc:2405] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:33.044647 14827 raft_consensus.cc:2743] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:33.084944 15015 raft_consensus.cc:695] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:33.089802 12719 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:33.091099 15051 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.088622 11968 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:33.181908 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.107995 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:32.238211 15068 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.084661 15093 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:33.162186 11893 raft_consensus.cc:2388] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 5: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:32.996007 15054 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:32.970053 15067 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.074s
I20250113 08:59:32.969237 15091 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.094s
W20250113 08:59:33.167196 12006 leader_election.cc:336] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 2.440s (SENT)
I20250113 08:59:33.087510 15101 raft_consensus.cc:2798] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:33.054288 14927 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:33.034222 14990 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:33.173411 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.193599 11963 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:33.194800 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.174784 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:33.013238 15108 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.188541 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.197382 11890 raft_consensus.cc:2405] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:33.198942 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.170174 14981 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 FOLLOWER]: Advancing to term 11
I20250113 08:59:33.195652 11967 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:33.199633 11889 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:33.200543 12006 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:33.195618 11963 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:33.220680 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:33.223227 12686 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.042s sys 0.099s
W20250113 08:59:33.225943 12686 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.042s sys 0.099s
I20250113 08:59:33.226711 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.227524 11968 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:33.229660 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.230326 11968 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:33.231455 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.198065 11964 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 8.
I20250113 08:59:33.233731 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 515) took 1003 ms. Trace:
I20250113 08:59:33.235404 11964 rpcz_store.cc:276] 0113 08:59:32.230738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.044392 (+813654us) service_pool.cc:225] Handling call
0113 08:59:33.233660 (+189268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:33.238168 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.238567 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.238991 11968 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:33.242547 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.246979 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:33.248347 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 430624us
I20250113 08:59:33.247107 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:33.249341 12725 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5.
I20250113 08:59:33.251026 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:33.251572 11934 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:33.252784 11934 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:33.256431 12684 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.009s sys 0.017s
W20250113 08:59:33.257203 12684 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.548s user 0.010s sys 0.017s
W20250113 08:59:33.260164 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 550911us
I20250113 08:59:33.228677 15027 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:33.230268 15045 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.240191 15112 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:33.271144 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.243036 15105 raft_consensus.cc:2743] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 08:59:33.241060 15051 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:33.304474 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:33.312551 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 203740us
I20250113 08:59:33.316576 12684 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:33.300711 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 344645us
I20250113 08:59:33.305606 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.303372 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:33.248415 15121 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:33.295642 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.252043 14827 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:33.259011 15058 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:33.333434 15058 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:33.333866 15058 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:33.333915 15106 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.258841 14729 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:33.334636 15106 consensus_queue.cc:1046] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:33.334945 14729 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:33.335361 14729 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:33.335666 15106 consensus_queue.cc:1035] T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:33.332916 14827 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:33.293581 15107 consensus_queue.cc:1046] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20250113 08:59:33.338305 14827 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:33.321264 14954 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.340471 12006 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:33.342638 15000 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.343333 15000 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:33.343935 15000 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:33.344560 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:33.344444 15000 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.345191 12725 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:33.346557 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:33.309846 15054 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.348210 11934 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:33.349177 15107 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:33.349581 15107 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:33.349952 15107 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:33.304293 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:33.311772 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:33.246476 15101 raft_consensus.cc:695] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:33.351941 15101 consensus_queue.cc:237] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:33.350630 15049 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.323922 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:33.355715 12038 raft_consensus.cc:2388] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:33.356518 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:33.357918 12038 raft_consensus.cc:1270] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250113 08:59:33.359181 12038 raft_consensus.cc:1270] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250113 08:59:33.221614 15093 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:33.361800 15120 log.cc:926] Time spent T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20250113 08:59:33.328600 15121 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.327425 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:33.363281 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 621) took 1959 ms. Trace:
I20250113 08:59:33.321019 15027 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.363744 11890 raft_consensus.cc:2388] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:33.294909 15015 consensus_queue.cc:237] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:33.364719 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:33.331395 12008 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:33.332274 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.366022 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.330118 15105 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:33.321451 14983 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.352193 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:33.361754 15093 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.320329 14455 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:33.366793 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.309661 14963 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:33.333011 15110 log.cc:926] Time spent T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
W20250113 08:59:33.368477 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 52039.7us
W20250113 08:59:33.359181 11858 leader_election.cc:336] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.830s (SENT)
I20250113 08:59:33.363770 15121 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:33.425683 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 66560.8us
W20250113 08:59:33.425994 15119 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
W20250113 08:59:33.426334 11858 leader_election.cc:336] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.147s (SENT)
I20250113 08:59:33.426795 11858 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:33.302877 15112 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.429636 15112 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.430397 14954 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250113 08:59:33.430963 15071 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.363649 11887 rpcz_store.cc:276] 0113 08:59:31.403547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.404209 (+ 662us) service_pool.cc:225] Handling call
0113 08:59:31.404679 (+ 470us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:31.709536 (+304857us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:31.709590 (+ 54us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:31.715330 (+ 5740us) log.cc:844] Serialized 52 byte log entry
0113 08:59:32.926776 (+1211446us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:32.926830 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:32.926922 (+ 92us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:33.363087 (+436165us) raft_consensus.cc:1680] finished
0113 08:59:33.363107 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:33.363223 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":175340,"raft.queue_time_us":790910,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":7362,"thread_start_us":1087941,"threads_started":1,"wal-append.queue_time_us":1306979}
W20250113 08:59:33.391736 15059 consensus_meta.cc:287] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.153s user 0.009s sys 0.001s
I20250113 08:59:33.385601 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:33.414004 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 82673.2us
I20250113 08:59:33.414256 12717 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:33.421325 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.435045 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2284) took 4682 ms. Trace:
I20250113 08:59:33.440045 11964 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:33.378545 11856 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:33.440220 11962 rpcz_store.cc:276] 0113 08:59:28.738005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.813903 (+ 75898us) service_pool.cc:225] Handling call
0113 08:59:28.834869 (+ 20966us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:29.013334 (+178465us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.419899 (+4406565us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54002a3d54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:33.420131 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":359440640}
I20250113 08:59:33.432621 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.421581 12685 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:33.434711 11858 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:33.436000 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 95607.9us
I20250113 08:59:33.376463 15027 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.446408 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.419154 15130 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:33.461040 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:33.436410 12007 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.001s sys 0.030s
I20250113 08:59:33.447559 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:33.455116 12006 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:33.415722 15045 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:33.455638 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:33.416702 14827 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:33.467680 11962 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [6.8-7.9]
W20250113 08:59:33.464452 12007 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.544s user 0.001s sys 0.030s
I20250113 08:59:33.468591 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:33.471794 12034 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250113 08:59:33.435432 15065 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:33.442318 15047 raft_consensus.cc:3049] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:33.324216 15051 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.442683 15021 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.445706 15123 raft_consensus.cc:988] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff: : Instructing follower 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:59:33.518610 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:33.476377 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 556204us
I20250113 08:59:33.433118 15132 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:33.481379 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2259) took 2114 ms. Trace:
I20250113 08:59:33.475075 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:33.518440 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:33.461048 14963 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:33.523012 11964 raft_consensus.cc:1270] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250113 08:59:33.562882 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 618) took 2541 ms. Trace:
I20250113 08:59:33.467159 15049 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:33.418916 15105 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.442978 15059 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.473644 15130 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:33.567560 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:33.809937 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2255) took 2811 ms. Trace:
W20250113 08:59:33.664261 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 136486us
W20250113 08:59:33.811874 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 602) took 3025 ms (client timeout 2037 ms). Trace:
I20250113 08:59:33.817327 11962 rpcz_store.cc:276] 0113 08:59:31.367016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.443253 (+2076237us) service_pool.cc:225] Handling call
0113 08:59:33.467429 (+ 24176us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:33.475680 (+ 8251us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.481303 (+ 5623us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:33.817546 12041 rpcz_store.cc:269] 0113 08:59:30.786382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:30.787322 (+ 940us) service_pool.cc:225] Handling call
0113 08:59:33.811623 (+3024301us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b54000cdad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:33.811735 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":613418880}
I20250113 08:59:33.816437 11965 rpcz_store.cc:276] 0113 08:59:31.019595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.137210 (+117615us) service_pool.cc:225] Handling call
0113 08:59:31.137495 (+ 285us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.161314 (+1023819us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.809691 (+1648377us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540029f254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:33.809876 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2353103232}
I20250113 08:59:33.818953 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:33.563408 11888 rpcz_store.cc:276] 0113 08:59:31.021801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.148965 (+127164us) service_pool.cc:225] Handling call
0113 08:59:31.179259 (+ 30294us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:31.415439 (+236180us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:31.415468 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:31.502331 (+ 86863us) log.cc:844] Serialized 26 byte log entry
0113 08:59:31.845978 (+343647us) raft_consensus.cc:1640] Marking committed up to 9
0113 08:59:31.846018 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.846091 (+ 73us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:33.562603 (+1716512us) raft_consensus.cc:1680] finished
0113 08:59:33.562622 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:33.562802 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38703,"raft.queue_time_us":106959,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":14855,"thread_start_us":427592,"threads_started":2,"wal-append.queue_time_us":1545460}
W20250113 08:59:33.816275 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 361234us
I20250113 08:59:33.814216 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.820156 12007 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:33.523332 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:33.754706 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2246) took 2785 ms (client timeout 2147 ms). Trace:
I20250113 08:59:33.825073 12725 raft_consensus.cc:2405] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:33.562461 11964 pending_rounds.cc:77] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting all ops after (but not including) 5
I20250113 08:59:33.827178 11964 pending_rounds.cc:99] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250113 08:59:33.827533 11964 raft_consensus.cc:2883] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:33.699676 11932 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.004s sys 0.012s
W20250113 08:59:33.828410 11932 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.012s user 0.005s sys 0.013s
W20250113 08:59:33.828781 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.81231e+06us
W20250113 08:59:33.734732 11966 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2248) took 2765 ms (client timeout 2354 ms). Trace:
I20250113 08:59:33.831049 12007 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:33.734175 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2270) took 7383 ms. Trace:
I20250113 08:59:33.478662 15045 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:33.831126 11966 rpcz_store.cc:269] 0113 08:59:30.969617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.734535 (+2764918us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:33.734650 (+ 115us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250113 08:59:33.572866 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 246077us
W20250113 08:59:33.824880 11961 rpcz_store.cc:269] 0113 08:59:30.969048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.754466 (+2785418us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:33.754621 (+ 155us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250113 08:59:33.972486 12037 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.005s sys 0.016s
W20250113 08:59:34.502166 12726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 806) took 3475 ms (client timeout 1798 ms). Trace:
I20250113 08:59:33.903793 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:33.832804 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 454355us
I20250113 08:59:33.853755 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 610) took 1510 ms. Trace:
W20250113 08:59:33.971024 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 147991us
W20250113 08:59:34.546008 12037 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.144s user 0.007s sys 0.033s
I20250113 08:59:33.844205 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.545495 11931 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:34.557394 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1882) took 1198 ms. Trace:
I20250113 08:59:34.557860 11888 rpcz_store.cc:276] 0113 08:59:33.359387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.843871 (+484484us) service_pool.cc:225] Handling call
0113 08:59:34.557333 (+713462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.558485 11931 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:33.997134 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.550211 12720 raft_consensus.cc:2388] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:59:33.811367 15053 consensus_meta.cc:287] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.803s user 0.006s sys 0.004s
W20250113 08:59:33.804143 15061 consensus_meta.cc:287] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.648s user 0.008s sys 0.003s
W20250113 08:59:33.726565 15031 consensus_meta.cc:287] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.541s user 0.007s sys 0.005s
W20250113 08:59:34.384392 15138 log.cc:926] Time spent T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
W20250113 08:59:34.613607 11962 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.013s sys 0.014s
W20250113 08:59:34.082144 12686 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.223s (SENT) [suppressed 35 similar messages]
W20250113 08:59:34.614444 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 532324us
I20250113 08:59:34.548228 11959 rpcz_store.cc:276] 0113 08:59:26.350903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:27.687708 (+1336805us) service_pool.cc:225] Handling call
0113 08:59:29.905324 (+2217616us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:30.653503 (+748179us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.733930 (+3080427us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b540006e0d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:33.734104 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5052172032}
I20250113 08:59:33.981595 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:33.992969 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.618054 12725 raft_consensus.cc:2371] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 5. Re-sending same reply.
I20250113 08:59:34.618470 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1879) took 1260 ms. Trace:
W20250113 08:59:34.171233 12039 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.006s sys 0.015s
W20250113 08:59:34.619624 12039 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.199s user 0.009s sys 0.015s
I20250113 08:59:34.619357 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.620623 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 596) took 5946 ms. Trace:
W20250113 08:59:34.552876 12726 rpcz_store.cc:269] 0113 08:59:31.026961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.030393 (+ 3432us) service_pool.cc:225] Handling call
0113 08:59:34.501970 (+3471577us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54006c9654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:34.502093 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3908705664}
I20250113 08:59:34.621086 12039 rpcz_store.cc:276] 0113 08:59:28.674586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:28.675149 (+ 563us) service_pool.cc:225] Handling call
0113 08:59:28.691197 (+ 16048us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:28.950602 (+259405us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:28.950637 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:28.951298 (+ 661us) log.cc:844] Serialized 52 byte log entry
0113 08:59:33.420277 (+4468979us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b18002044c0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336b2e55 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336d08c2 00007fc336285e4b 00007fc33628b913 00007fc33628bcc1 00007fc3365dd45f
0113 08:59:34.620100 (+1199823us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.620140 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.620218 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:34.620269 (+ 51us) raft_consensus.cc:1680] finished
0113 08:59:34.620286 (+ 17us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.620537 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6912,"raft.queue_time_us":21296,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":3530,"spinlock_wait_cycles":14488960,"thread_start_us":1199601,"threads_started":1,"wal-append.queue_time_us":5630093}
I20250113 08:59:34.622749 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.623055 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:34.497016 14729 consensus_meta.cc:287] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.161s user 0.010s sys 0.002s
I20250113 08:59:34.623754 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 873) took 1177 ms. Trace:
I20250113 08:59:34.195390 15157 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:34.624272 12726 rpcz_store.cc:276] 0113 08:59:33.446764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.622423 (+1175659us) service_pool.cc:225] Handling call
0113 08:59:34.623675 (+ 1252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:33.930337 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:34.625052 15157 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.625687 12038 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:34.626360 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 615) took 1182 ms. Trace:
I20250113 08:59:34.626739 12038 rpcz_store.cc:276] 0113 08:59:33.443543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.929938 (+486395us) service_pool.cc:225] Handling call
0113 08:59:34.626291 (+696353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.316506 15059 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.847306 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:34.629335 15059 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250113 08:59:34.455391 15149 log.cc:926] Time spent T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.433s user 0.000s sys 0.001s
I20250113 08:59:34.629966 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 618) took 1112 ms. Trace:
I20250113 08:59:34.630481 12034 rpcz_store.cc:276] 0113 08:59:33.517747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.846989 (+329242us) service_pool.cc:225] Handling call
0113 08:59:34.629893 (+782904us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.552723 12684 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:33.806071 15073 consensus_meta.cc:287] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.841s user 0.005s sys 0.005s
I20250113 08:59:34.456350 15130 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:34.631407 15073 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.631635 15130 consensus_queue.cc:1035] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:34.632489 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 667) took 1379 ms. Trace:
I20250113 08:59:34.021816 15151 consensus_queue.cc:1035] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.643s
I20250113 08:59:34.633870 15073 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:34.632905 12721 rpcz_store.cc:276] 0113 08:59:33.252706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.253201 (+ 495us) service_pool.cc:225] Handling call
0113 08:59:33.293946 (+ 40745us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:33.367871 (+ 73925us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:33.367905 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:33.382262 (+ 14357us) log.cc:844] Serialized 26 byte log entry
0113 08:59:33.606822 (+224560us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:33.606877 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.606991 (+ 114us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:34.632219 (+1025228us) raft_consensus.cc:1680] finished
0113 08:59:34.632244 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.632426 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73450,"raft.queue_time_us":73787,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":36978,"thread_start_us":140716,"threads_started":1,"wal-append.queue_time_us":376708}
W20250113 08:59:33.964687 15058 consensus_meta.cc:287] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.630s user 0.009s sys 0.002s
W20250113 08:59:34.500735 11932 connection.cc:232] RPC call timeout handler was delayed by -1.3223643720000382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
W20250113 08:59:34.634780 11932 connection.cc:232] RPC call timeout handler was delayed by -1.2578487040000255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1585s for a response
I20250113 08:59:34.634558 15058 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:34.635140 11932 connection.cc:232] RPC call timeout handler was delayed by -1.2266034320000472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20250113 08:59:33.987638 11858 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.011s sys 0.012s
W20250113 08:59:34.635460 11932 connection.cc:232] RPC call timeout handler was delayed by -1.22210343200004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20250113 08:59:34.635581 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.192s user 0.014s sys 0.022s
I20250113 08:59:34.635808 11963 raft_consensus.cc:2388] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:34.635964 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.20126e+06us
W20250113 08:59:33.737322 15083 log.cc:926] Time spent T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 2.330s user 0.008s sys 0.004s
I20250113 08:59:34.459640 15105 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.903896 11965 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250113 08:59:33.800011 15026 log.cc:926] Time spent T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 4.455s user 0.017s sys 0.064s
I20250113 08:59:33.635604 15065 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:34.638017 14798 raft_consensus.cc:2743] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:34.562649 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 875) took 1092 ms. Trace:
I20250113 08:59:34.638316 15065 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:33.567541 15126 raft_consensus.cc:2743] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:34.638640 12720 rpcz_store.cc:276] 0113 08:59:33.470001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.903530 (+433529us) service_pool.cc:225] Handling call
0113 08:59:34.562578 (+659048us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.621013 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:34.171324 15054 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.004s sys 0.009s
I20250113 08:59:34.639895 12725 raft_consensus.cc:2388] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
W20250113 08:59:34.640056 15054 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.012s user 0.008s sys 0.018s
I20250113 08:59:34.640350 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 877) took 1115 ms. Trace:
I20250113 08:59:33.711627 14966 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.820644 15132 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:34.640667 12725 rpcz_store.cc:276] 0113 08:59:33.524768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.620674 (+1095906us) service_pool.cc:225] Handling call
0113 08:59:34.640295 (+ 19621us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:33.482542 15047 raft_consensus.cc:738] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:34.570986 15129 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:34.641650 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.641992 15047 consensus_queue.cc:260] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.642192 15129 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.586082 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.555436 12036 rpcz_store.cc:276] 0113 08:59:32.342931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.039613 (+696682us) service_pool.cc:225] Handling call
0113 08:59:33.059434 (+ 19821us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:33.109012 (+ 49578us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:33.109049 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:33.171980 (+ 62931us) log.cc:844] Serialized 52 byte log entry
0113 08:59:33.449343 (+277363us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:33.471371 (+ 22028us) log.cc:844] Serialized 14 byte log entry
0113 08:59:33.853236 (+381865us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.853396 (+ 160us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:33.853458 (+ 62us) raft_consensus.cc:1680] finished
0113 08:59:33.853477 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:33.853677 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40819,"prepare.queue_time_us":17121,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":291132,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":25259,"thread_start_us":617694,"threads_started":2,"wal-append.queue_time_us":474279}
I20250113 08:59:34.643227 15047 raft_consensus.cc:3054] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:33.735355 14950 raft_consensus.cc:3049] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:34.639889 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.643821 14950 raft_consensus.cc:738] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:34.644280 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 882) took 1076 ms. Trace:
I20250113 08:59:34.644662 14950 consensus_queue.cc:260] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:34.355463 15121 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.010s sys 0.008s
W20250113 08:59:34.635813 11932 connection.cc:232] RPC call timeout handler was delayed by -1.115494281999986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18830000000000002s for a response
I20250113 08:59:34.645771 14950 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:34.646083 15121 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.012s user 0.014s sys 0.011s
I20250113 08:59:33.808126 15107 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.647810 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.648415 12726 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:33.753862 14783 raft_consensus.cc:3049] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:34.644280 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.649456 14783 raft_consensus.cc:738] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:34.649802 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 519) took 2418 ms. Trace:
I20250113 08:59:34.650563 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:34.650471 14783 consensus_queue.cc:260] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.587882 15061 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:34.651907 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:34.652228 11959 raft_consensus.cc:1235] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:34.650189 11965 rpcz_store.cc:276] 0113 08:59:32.231745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.819199 (+1587454us) service_pool.cc:225] Handling call
0113 08:59:33.903536 (+ 84337us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.649533 (+745997us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.649746 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.653558 11960 raft_consensus.cc:2363] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8.
I20250113 08:59:34.653467 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:34.654817 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:34.655185 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 797) took 5494 ms. Trace:
I20250113 08:59:34.656080 11965 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:34.656899 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2260) took 1691 ms. Trace:
I20250113 08:59:33.523522 14990 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.568702 14954 raft_consensus.cc:1076] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Signalling peer 5e8883f5f83a47f4b9303dd1da4f59c0 to start an election
I20250113 08:59:33.733711 15000 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.585062 14963 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:34.196182 15051 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:34.590927 15053 consensus_queue.cc:1046] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:34.591728 15146 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:34.623839 14729 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:33.959008 14865 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.588s user 0.023s sys 0.041s
W20250113 08:59:33.945513 15001 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 2.545s user 0.009s sys 0.022s
I20250113 08:59:34.592869 15144 consensus_queue.cc:1046] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250113 08:59:34.586158 15045 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:34.593393 15031 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:34.596796 12006 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:34.649127 15107 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:33.844144 15041 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:34.605793 15143 log.cc:926] Time spent T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.228s user 0.000s sys 0.000s
I20250113 08:59:34.606998 15040 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:34.607718 14976 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:34.655570 12724 rpcz_store.cc:276] 0113 08:59:29.160839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:29.160995 (+ 156us) service_pool.cc:225] Handling call
0113 08:59:29.828438 (+667443us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:30.646035 (+817597us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:30.782860 (+136825us) log.cc:844] Serialized 14 byte log entry
0113 08:59:31.084883 (+302023us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:31.085437 (+ 554us) log.cc:844] Serialized 26 byte log entry
0113 08:59:31.085588 (+ 151us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:31.085682 (+ 94us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:31.085838 (+ 156us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:33.736781 (+2650943us) spinlock_profiling.cc:243] Waited 68.4 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:34.654858 (+918077us) raft_consensus.cc:1680] finished
0113 08:59:34.654991 (+ 133us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.655124 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14633,"prepare.queue_time_us":268299,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":200978,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":186,"spinlock_wait_cycles":157404416,"thread_start_us":1145642,"threads_started":3,"wal-append.queue_time_us":394415}
I20250113 08:59:33.968680 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:34.631757 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 79111.7us
I20250113 08:59:34.641161 15132 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:34.634974 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.561623 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1875) took 1255 ms. Trace:
I20250113 08:59:33.965500 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 511) took 1736 ms. Trace:
W20250113 08:59:33.643034 14845 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 4.292s user 0.055s sys 0.060s
I20250113 08:59:34.636135 11968 raft_consensus.cc:2388] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:34.638217 11858 leader_election.cc:336] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.354s (SENT)
I20250113 08:59:34.569554 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2293) took 2222 ms. Trace:
I20250113 08:59:34.637080 15105 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:34.643000 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.651160 15065 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:34.653015 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:33.904104 14937 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:34.644649 11887 raft_consensus.cc:2405] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:34.594674 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 606) took 3113 ms. Trace:
I20250113 08:59:34.653191 15061 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:33.596405 15015 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250113 08:59:34.659358 15053 consensus_queue.cc:1225] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:34.698328 12726 raft_consensus.cc:2405] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:34.657227 11965 rpcz_store.cc:276] 0113 08:59:32.965707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.655644 (+1689937us) service_pool.cc:225] Handling call
0113 08:59:34.655917 (+ 273us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.656678 (+ 761us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.656849 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.676703 12042 rpcz_store.cc:276] 0113 08:59:31.481875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.985166 (+1503291us) service_pool.cc:225] Handling call
0113 08:59:32.985658 (+ 492us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:33.353725 (+368067us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:33.353781 (+ 56us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:33.354345 (+ 564us) log.cc:844] Serialized 26 byte log entry
0113 08:59:33.825451 (+471106us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:33.825512 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.825620 (+ 108us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:34.594449 (+768829us) raft_consensus.cc:1680] finished
0113 08:59:34.594474 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.594618 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15864,"raft.queue_time_us":55362,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":23059,"thread_start_us":720158,"threads_started":2,"wal-append.queue_time_us":1145773}
I20250113 08:59:34.723294 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 881) took 1155 ms. Trace:
I20250113 08:59:34.724241 12684 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:34.688477 11962 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.013s sys 0.014s
I20250113 08:59:34.687258 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.726344 12725 raft_consensus.cc:2388] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:34.692762 12039 raft_consensus.cc:2405] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 9 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250113 08:59:34.675848 11966 rpcz_store.cc:276] 0113 08:59:32.346990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.545625 (+2198635us) service_pool.cc:225] Handling call
0113 08:59:34.568932 (+ 23307us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.569161 (+ 229us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:34.569197 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:34.569267 (+ 70us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.569303 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.569355 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.569475 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.676888 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1881) took 1317 ms. Trace:
I20250113 08:59:34.673604 12723 raft_consensus.cc:2405] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250113 08:59:34.696643 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 108725us
I20250113 08:59:34.659997 14990 consensus_queue.cc:1046] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20250113 08:59:34.660082 14963 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:34.660391 15001 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 3.260s user 0.018s sys 0.028s
I20250113 08:59:34.741320 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 620) took 1156 ms. Trace:
I20250113 08:59:34.741717 12039 rpcz_store.cc:276] 0113 08:59:33.585006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.622809 (+1037803us) service_pool.cc:225] Handling call
0113 08:59:34.741238 (+118429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.661432 14966 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 08:59:34.663290 15041 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:34.661500 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 64785.3us
I20250113 08:59:34.704200 14729 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:34.662997 15031 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:34.667407 11889 rpcz_store.cc:276] 0113 08:59:33.306697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.903173 (+596476us) service_pool.cc:225] Handling call
0113 08:59:34.561562 (+658389us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:34.745241 12040 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.268s user 0.012s sys 0.031s
I20250113 08:59:34.644647 12720 rpcz_store.cc:276] 0113 08:59:33.567823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.639562 (+1071739us) service_pool.cc:225] Handling call
0113 08:59:34.644220 (+ 4658us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.675663 12008 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:34.746186 12040 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.269s user 0.013s sys 0.031s
I20250113 08:59:34.663398 15107 consensus_queue.cc:1046] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20250113 08:59:34.670092 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.676450 14937 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:34.725116 12726 rpcz_store.cc:276] 0113 08:59:33.567555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.625187 (+1057632us) service_pool.cc:225] Handling call
0113 08:59:34.723214 (+ 98027us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.668411 11964 rpcz_store.cc:276] 0113 08:59:32.229583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.476659 (+1247076us) service_pool.cc:225] Handling call
0113 08:59:33.522672 (+ 46013us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:33.965093 (+442421us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:33.965404 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.747032 14937 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:34.747674 12724 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:34.748494 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.749881 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.750515 12726 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:34.731351 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 876) took 1252 ms. Trace:
I20250113 08:59:34.754331 12725 rpcz_store.cc:276] 0113 08:59:33.478629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.686894 (+1208265us) service_pool.cc:225] Handling call
0113 08:59:34.731283 (+ 44389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.755487 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.756994 12725 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:34.724148 11965 raft_consensus.cc:1235] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20250113 08:59:34.692454 15047 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:34.741164 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:34.701122 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1750) took 2807 ms. Trace:
I20250113 08:59:34.759829 15047 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:34.619022 11890 rpcz_store.cc:276] 0113 08:59:33.358495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.903953 (+545458us) service_pool.cc:225] Handling call
0113 08:59:34.618389 (+714436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.761912 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.667770 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 884) took 1064 ms. Trace:
I20250113 08:59:34.730049 11888 rpcz_store.cc:276] 0113 08:59:33.359088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.652790 (+1293702us) service_pool.cc:225] Handling call
0113 08:59:34.676817 (+ 24027us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.762950 11890 raft_consensus.cc:2405] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:34.668601 14845 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 5.318s user 0.073s sys 0.086s
I20250113 08:59:34.763125 12721 rpcz_store.cc:276] 0113 08:59:33.603070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.634671 (+1031601us) service_pool.cc:225] Handling call
0113 08:59:34.667698 (+ 33027us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:34.693960 15044 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.013s sys 0.023s
W20250113 08:59:34.660291 14865 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 3.293s user 0.029s sys 0.058s
I20250113 08:59:34.759879 12037 rpcz_store.cc:276] 0113 08:59:31.894121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.043491 (+1149370us) service_pool.cc:225] Handling call
0113 08:59:33.047105 (+ 3614us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:33.047305 (+ 200us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:33.402164 (+354859us) log.cc:844] Serialized 14 byte log entry
0113 08:59:34.700773 (+1298609us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:34.700829 (+ 56us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.700880 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.700947 (+ 67us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.701068 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4206,"prepare.queue_time_us":290365,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":79851,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":1648357,"threads_started":3}
I20250113 08:59:34.677428 15015 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:34.761999 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2261) took 1795 ms. Trace:
I20250113 08:59:34.770429 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.793661 11965 rpcz_store.cc:276] 0113 08:59:32.966541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.723449 (+1756908us) service_pool.cc:225] Handling call
0113 08:59:34.723815 (+ 366us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:34.760055 (+ 36240us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.761694 (+ 1639us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.794119 11889 raft_consensus.cc:2405] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 8 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:34.794752 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:34.795703 11965 raft_consensus.cc:2388] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:34.771318 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 940176us
W20250113 08:59:34.771966 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 96367.7us
I20250113 08:59:34.730986 15058 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:34.627590 15157 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:34.774251 12724 raft_consensus.cc:2463] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:34.805974 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.806172 11934 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
W20250113 08:59:34.782295 15044 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s user 0.014s sys 0.023s
I20250113 08:59:34.807090 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.808344 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.660741 15144 consensus_queue.cc:1225] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250113 08:59:34.773924 12726 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:34.762454 15047 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:34.676052 15105 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:34.660720 15059 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:34.730216 14994 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:34.786679 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.842911 11888 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:34.790954 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.677374 15093 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:34.772976 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 631) took 1452 ms. Trace:
I20250113 08:59:34.795850 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.764936 14954 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:34.813604 12726 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:34.852638 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 814) took 1411 ms. Trace:
I20250113 08:59:34.845880 12725 raft_consensus.cc:2463] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:34.814169 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:34.696871 14950 consensus_queue.cc:1046] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.102s
I20250113 08:59:34.821045 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.785598 14865 raft_consensus.cc:695] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:34.797168 14940 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:34.820441 11934 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:34.844925 11890 rpcz_store.cc:276] 0113 08:59:33.320805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.761619 (+1440814us) service_pool.cc:225] Handling call
0113 08:59:34.772917 (+ 11298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.846774 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 878) took 1321 ms. Trace:
I20250113 08:59:34.853091 12726 rpcz_store.cc:276] 0113 08:59:33.442751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.813083 (+1370332us) service_pool.cc:225] Handling call
0113 08:59:34.813379 (+ 296us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.852375 (+ 38996us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.852582 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:34.700156 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 141760us
I20250113 08:59:34.688342 15065 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.858610 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:34.862102 12721 rpcz_store.cc:276] 0113 08:59:33.525129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.770062 (+1244933us) service_pool.cc:225] Handling call
0113 08:59:34.846699 (+ 76637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.840320 15105 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:34.870527 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 50153.5us
W20250113 08:59:34.859272 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 209916us
I20250113 08:59:34.858227 12725 raft_consensus.cc:1270] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250113 08:59:34.885663 11934 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:34.886162 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 872) took 1439 ms. Trace:
I20250113 08:59:34.886713 12725 rpcz_store.cc:276] 0113 08:59:33.446434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.856018 (+1409584us) service_pool.cc:225] Handling call
0113 08:59:34.856506 (+ 488us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.885449 (+ 28943us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.886090 (+ 641us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.842705 15059 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:34.887005 12724 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:34.889374 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1871) took 1806 ms. Trace:
I20250113 08:59:34.802160 15001 consensus_queue.cc:1046] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20250113 08:59:34.805228 14966 consensus_queue.cc:1035] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
I20250113 08:59:34.852284 14954 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:34.907521 14990 raft_consensus.cc:2798] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:34.907699 14954 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:34.908150 14990 raft_consensus.cc:695] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:34.893887 11932 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:34.909844 11932 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:34.894378 11889 rpcz_store.cc:276] 0113 08:59:33.083482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.820804 (+1737322us) service_pool.cc:225] Handling call
0113 08:59:34.889292 (+ 68488us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.910701 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 623) took 2362 ms. Trace:
I20250113 08:59:34.894639 11931 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:34.897333 12721 raft_consensus.cc:1235] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250113 08:59:34.855063 14994 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.857134 15159 raft_consensus.cc:2798] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:34.893569 14729 raft_consensus.cc:2798] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:34.902549 15001 consensus_queue.cc:1225] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250113 08:59:34.906059 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 815) took 1460 ms. Trace:
I20250113 08:59:34.906733 12725 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:34.928376 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 817) took 1451 ms. Trace:
I20250113 08:59:34.911227 11890 rpcz_store.cc:276] 0113 08:59:32.548859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.880438 (+2331579us) service_pool.cc:225] Handling call
0113 08:59:34.909238 (+ 28800us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.910296 (+ 1058us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:34.910330 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:34.910376 (+ 46us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.910429 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.910519 (+ 90us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.910648 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.912568 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 874) took 1443 ms. Trace:
I20250113 08:59:34.908934 15065 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:34.895421 14950 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:34.913393 14990 consensus_queue.cc:237] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.802794 15015 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:34.916108 11932 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:34.917776 11931 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:34.919740 11889 raft_consensus.cc:1270] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250113 08:59:34.929069 12725 rpcz_store.cc:276] 0113 08:59:33.477070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.888077 (+1411007us) service_pool.cc:225] Handling call
0113 08:59:34.906487 (+ 18410us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.928065 (+ 21578us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.928305 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.900928 15058 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20250113 08:59:34.930408 12721 rpcz_store.cc:276] 0113 08:59:33.469581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.869526 (+1399945us) service_pool.cc:225] Handling call
0113 08:59:34.897000 (+ 27474us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:34.912302 (+ 15302us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.912512 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.868434 15144 consensus_queue.cc:1046] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20250113 08:59:34.938592 12724 rpcz_store.cc:276] 0113 08:59:33.446075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.855098 (+1409023us) service_pool.cc:225] Handling call
0113 08:59:34.871056 (+ 15958us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.895426 (+ 24370us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.905982 (+ 10556us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.855787 15093 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:34.940909 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 209309us
I20250113 08:59:34.946113 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 626) took 2370 ms. Trace:
I20250113 08:59:34.956732 11889 rpcz_store.cc:276] 0113 08:59:32.575711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.911259 (+2335548us) service_pool.cc:225] Handling call
0113 08:59:34.919289 (+ 8030us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.945790 (+ 26501us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.946052 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.954176 11932 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:34.952003 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2303) took 3476 ms. Trace:
I20250113 08:59:34.915753 14940 raft_consensus.cc:695] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:34.856348 15171 consensus_queue.cc:1046] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20250113 08:59:34.920861 15073 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:34.926548 15040 raft_consensus.cc:2798] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:34.969971 11890 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: []
W20250113 08:59:34.958920 12686 leader_election.cc:336] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.504s (SENT)
I20250113 08:59:34.971448 12686 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:34.959196 12040 rpcz_store.cc:276] 0113 08:59:31.475611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.963888 (+1488277us) service_pool.cc:225] Handling call
0113 08:59:32.964194 (+ 306us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.964461 (+ 267us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:33.300172 (+335711us) log.cc:844] Serialized 14 byte log entry
0113 08:59:34.949003 (+1648831us) log.cc:844] Serialized 14 byte log entry
0113 08:59:34.951404 (+ 2401us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:34.951451 (+ 47us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.951501 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.951560 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.951943 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6461,"prepare.queue_time_us":496164,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":4413,"raft.queue_time_us":188818,"raft.run_cpu_time_us":350,"raft.run_wall_time_us":5938,"thread_start_us":1931010,"threads_started":4,"wal-append.queue_time_us":683435}
I20250113 08:59:34.966404 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:34.974246 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 818) took 1491 ms. Trace:
I20250113 08:59:34.974673 12725 rpcz_store.cc:276] 0113 08:59:33.483372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.959566 (+1476194us) service_pool.cc:225] Handling call
0113 08:59:34.973696 (+ 14130us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.973924 (+ 228us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:34.973953 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:34.973996 (+ 43us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.974030 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.974080 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.974194 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.951050 15159 raft_consensus.cc:695] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:34.955869 15031 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:35.038597 12721 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20250113 08:59:35.048735 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 879) took 1523 ms. Trace:
I20250113 08:59:35.049225 12721 rpcz_store.cc:276] 0113 08:59:33.525472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.973710 (+1448238us) service_pool.cc:225] Handling call
0113 08:59:35.038306 (+ 64596us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:35.048374 (+ 10068us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.048663 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.976611 15065 raft_consensus.cc:3049] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:34.996604 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 816) took 1528 ms. Trace:
I20250113 08:59:34.980489 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 627) took 2390 ms. Trace:
I20250113 08:59:35.054224 11890 rpcz_store.cc:276] 0113 08:59:32.580821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.944469 (+2363648us) service_pool.cc:225] Handling call
0113 08:59:34.969772 (+ 25303us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.970468 (+ 696us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.970663 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:34.969563 15073 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:34.969871 15040 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:34.966039 15180 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:34.950366 14950 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:35.052172 15138 log.cc:926] Time spent T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250113 08:59:35.039165 15173 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:35.073173 11890 raft_consensus.cc:1270] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250113 08:59:35.074060 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1858) took 2125 ms. Trace:
I20250113 08:59:35.060431 15073 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:35.077201 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 118302us
I20250113 08:59:35.077781 12725 raft_consensus.cc:1235] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:35.078236 15093 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:35.078451 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 880) took 1511 ms. Trace:
I20250113 08:59:34.970189 14865 consensus_queue.cc:237] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.081540 15041 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:35.074496 11890 rpcz_store.cc:276] 0113 08:59:32.948669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.055493 (+2106824us) service_pool.cc:225] Handling call
0113 08:59:35.072792 (+ 17299us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.073715 (+ 923us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.073986 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:35.078853 12725 rpcz_store.cc:276] 0113 08:59:33.567224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.975774 (+1408550us) service_pool.cc:225] Handling call
0113 08:59:35.077532 (+101758us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:35.078196 (+ 664us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.078399 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:35.028012 15144 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
I20250113 08:59:35.064440 15040 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:35.053890 15065 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:35.054108 12726 rpcz_store.cc:276] 0113 08:59:33.468133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.862354 (+1394221us) service_pool.cc:225] Handling call
0113 08:59:34.862679 (+ 325us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.862966 (+ 287us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:34.863276 (+ 310us) log.cc:844] Serialized 14 byte log entry
0113 08:59:34.994549 (+131273us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:34.994626 (+ 77us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.994691 (+ 65us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.994792 (+ 101us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:34.996511 (+ 1719us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21566,"prepare.queue_time_us":6393,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":109219,"threads_started":1}
I20250113 08:59:35.087711 12726 raft_consensus.cc:1235] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:35.088470 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 672) took 1370 ms. Trace:
I20250113 08:59:35.079013 15019 raft_consensus.cc:3049] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:35.091758 15019 raft_consensus.cc:738] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.092900 11962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 522) took 2860 ms. Trace:
I20250113 08:59:35.093021 15019 consensus_queue.cc:260] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.094035 15177 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 08:59:35.081714 14963 raft_consensus.cc:3049] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:35.088876 12726 rpcz_store.cc:276] 0113 08:59:33.718165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.087064 (+1368899us) service_pool.cc:225] Handling call
0113 08:59:35.087468 (+ 404us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.088194 (+ 726us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.088409 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:35.084209 12686 leader_election.cc:336] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.548s (SENT)
I20250113 08:59:35.102034 12686 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.094458 15019 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:35.112787 15170 log.cc:926] Time spent T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.162s user 0.001s sys 0.001s
I20250113 08:59:35.113670 15054 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:34.966516 15172 consensus_queue.cc:1046] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20250113 08:59:35.093402 11962 rpcz_store.cc:276] 0113 08:59:32.232623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.819828 (+1587205us) service_pool.cc:225] Handling call
0113 08:59:33.947027 (+127199us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:33.955454 (+ 8427us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:33.955506 (+ 52us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:33.956028 (+ 522us) log.cc:844] Serialized 26 byte log entry
0113 08:59:34.726091 (+770063us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:34.951046 (+224955us) log.cc:844] Serialized 14 byte log entry
0113 08:59:34.951250 (+ 204us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.951325 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:35.092523 (+141198us) raft_consensus.cc:1680] finished
0113 08:59:35.092544 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:35.092692 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23080,"prepare.queue_time_us":129414,"prepare.run_cpu_time_us":706,"prepare.run_wall_time_us":12683,"raft.queue_time_us":8066,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":24235,"thread_start_us":889376,"threads_started":2,"wal-append.queue_time_us":1117455}
W20250113 08:59:35.119813 15163 log.cc:926] Time spent T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.220s user 0.001s sys 0.000s
I20250113 08:59:35.120664 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:35.122061 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:35.122767 11962 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:35.061012 15132 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.094973 15041 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250113 08:59:35.125084 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 171000us
I20250113 08:59:35.130235 12007 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:35.099112 14963 raft_consensus.cc:738] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.083017 14940 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.079005 15093 consensus_queue.cc:1035] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20250113 08:59:35.135892 12726 raft_consensus.cc:1235] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:35.136323 12041 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:35.134058 12721 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20250113 08:59:35.138942 14966 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:35.110472 15159 consensus_queue.cc:237] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.140898 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 883) took 1538 ms. Trace:
I20250113 08:59:35.109508 15065 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.141402 12721 rpcz_store.cc:276] 0113 08:59:33.602695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.082932 (+1480237us) service_pool.cc:225] Handling call
0113 08:59:35.133559 (+ 50627us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:35.140521 (+ 6962us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.140820 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:35.123844 15179 consensus_queue.cc:1046] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20250113 08:59:35.131778 11932 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.151800 11932 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:35.156785 15155 log.cc:926] Time spent T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.266s user 0.001s sys 0.001s
I20250113 08:59:35.153296 15177 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:35.160204 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2298) took 2345 ms. Trace:
I20250113 08:59:35.160622 11961 rpcz_store.cc:276] 0113 08:59:32.814987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.549545 (+1734558us) service_pool.cc:225] Handling call
0113 08:59:34.586549 (+ 37004us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:34.694229 (+107680us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:34.694260 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:34.695143 (+ 883us) log.cc:844] Serialized 26 byte log entry
0113 08:59:34.937531 (+242388us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:34.937582 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.937698 (+ 116us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:35.159981 (+222283us) raft_consensus.cc:1680] finished
0113 08:59:35.160003 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:35.160144 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":47575,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":348097,"threads_started":2,"wal-append.queue_time_us":457812}
I20250113 08:59:35.135113 15150 consensus_queue.cc:1046] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20250113 08:59:35.164155 14950 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:35.164768 15132 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:35.155728 11932 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.170063 11932 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.172240 12721 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:35.160553 15173 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:35.181982 14621 consensus_queue.cc:1035] T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:35.186053 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:35.184365 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 886) took 1211 ms. Trace:
I20250113 08:59:35.186879 12721 rpcz_store.cc:276] 0113 08:59:33.965928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.147196 (+1181268us) service_pool.cc:225] Handling call
0113 08:59:35.168505 (+ 21309us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.176870 (+ 8365us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.177191 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:35.170629 15015 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:35.191280 15187 consensus_queue.cc:1035] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20250113 08:59:35.191627 15041 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:35.187290 15044 raft_consensus.cc:2743] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250113 08:59:35.195017 14783 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:35.197654 15183 log.cc:926] Time spent T badac2289c8b4a10915cca20182fc423 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20250113 08:59:35.173436 15177 raft_consensus.cc:2743] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:35.205235 11962 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:35.221731 14455 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:35.228314 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:35.228782 14455 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.229630 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:35.230684 15184 log.cc:926] Time spent T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20250113 08:59:35.231006 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:35.231276 14455 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:35.232177 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:35.233836 11858 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:35.185164 11932 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.183457 15033 consensus_queue.cc:1046] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.028s
I20250113 08:59:35.214768 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:35.235183 12721 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.236399 11962 raft_consensus.cc:2405] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
W20250113 08:59:35.237444 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 52369.5us
I20250113 08:59:35.239135 11932 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:35.242683 15066 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:59:35.245954 15033 consensus_queue.cc:1225] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:35.253005 15066 raft_consensus.cc:695] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 8 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:35.225380 15041 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.256254 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1743) took 4167 ms. Trace:
I20250113 08:59:35.256803 12033 rpcz_store.cc:276] 0113 08:59:31.089339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.121898 (+ 32559us) service_pool.cc:225] Handling call
0113 08:59:31.122161 (+ 263us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:32.157793 (+1035632us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.255984 (+3098191us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540002cbd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:35.256176 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2380428544}
I20250113 08:59:35.260757 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:35.245895 15060 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:35.171809 14963 consensus_queue.cc:260] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.241376 15044 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:35.266709 15041 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:35.263947 15060 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:35.293135 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 265070us
I20250113 08:59:35.300740 11856 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:35.308931 15067 consensus_queue.cc:1046] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:35.309993 15067 consensus_queue.cc:1225] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20250113 08:59:35.309397 15043 log.cc:926] Time spent T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 4.668s user 0.007s sys 0.022s
W20250113 08:59:35.314945 12686 leader_election.cc:336] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.881s (SENT)
I20250113 08:59:35.315559 12686 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.316550 15041 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:35.319402 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.320175 12040 raft_consensus.cc:1235] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:35.320276 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.321231 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.321880 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.322641 12036 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:35.321209 12033 raft_consensus.cc:2405] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20250113 08:59:35.325525 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 517) took 3094 ms (client timeout 1548 ms). Trace:
W20250113 08:59:35.326025 11967 rpcz_store.cc:269] 0113 08:59:32.231248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.900745 (+669497us) service_pool.cc:225] Handling call
0113 08:59:35.325275 (+2424530us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54001a1dd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:35.325435 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":484284288}
I20250113 08:59:35.327306 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:35.331063 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.331614 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.331775 12034 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.333161 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.334307 12686 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:35.335122 11886 raft_consensus.cc:3049] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:35.335481 11886 raft_consensus.cc:738] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:35.335755 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.327960 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 819) took 1804 ms. Trace:
I20250113 08:59:35.283795 15060 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:35.336617 12724 rpcz_store.cc:276] 0113 08:59:33.523825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.950920 (+1427095us) service_pool.cc:225] Handling call
0113 08:59:35.327261 (+376341us) spinlock_profiling.cc:243] Waited 5.35 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:59:35.327451 (+ 190us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.327647 (+ 196us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:35.327671 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:35.327712 (+ 41us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:35.327742 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.327795 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:35.327888 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12315392}
I20250113 08:59:35.344581 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.345309 12033 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.345384 12686 leader_election.cc:304] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.350051 12686 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:35.305394 14950 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:35.373584 12037 raft_consensus.cc:1235] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:35.352902 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:35.353273 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.374631 11967 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:35.359440 11886 consensus_queue.cc:260] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.348299 14621 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.003s
I20250113 08:59:35.254354 15066 consensus_queue.cc:237] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.385448 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:35.322760 14717 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:35.351788 15041 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:35.387938 11886 raft_consensus.cc:3054] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.395893 12721 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:35.372303 15033 consensus_queue.cc:1035] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
W20250113 08:59:35.383124 14937 consensus_meta.cc:287] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.601s user 0.006s sys 0.005s
I20250113 08:59:35.399622 14950 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:35.400221 14950 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.384594 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.382894 15172 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:35.398607 12008 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:35.402076 12037 raft_consensus.cc:3054] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.403522 12036 raft_consensus.cc:2405] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250113 08:59:35.410269 11856 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:35.404083 11931 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:35.415973 12033 raft_consensus.cc:2463] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:35.423115 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.424027 12036 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:35.424340 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.425379 12039 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:35.395547 15041 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:35.451717 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.384330 15073 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:35.461397 12684 leader_election.cc:336] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231): Timed out: RequestConsensusVote RPC to 127.10.115.131:43231 timed out after 2.037s (SENT)
I20250113 08:59:35.418452 15044 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:35.409922 14717 consensus_queue.cc:1046] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.064s
I20250113 08:59:35.450582 15053 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250113 08:59:35.488154 12034 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
W20250113 08:59:35.460935 15000 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.026s sys 0.011s
I20250113 08:59:35.489214 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.518983 12034 raft_consensus.cc:2405] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250113 08:59:35.520197 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.520941 12034 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:35.456413 15041 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.524724 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.408375 14937 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.512972 15044 raft_consensus.cc:3049] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:35.495153 12037 raft_consensus.cc:2463] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:35.445075 14729 raft_consensus.cc:695] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:35.525779 15100 log.cc:926] Time spent T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 3.090s user 0.012s sys 0.012s
I20250113 08:59:35.525620 15030 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:35.526178 15109 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:35.526105 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.429885 15005 consensus_queue.cc:1046] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
W20250113 08:59:35.530488 14967 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250113 08:59:35.530484 12036 raft_consensus.cc:2463] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:35.534693 15190 raft_consensus.cc:2743] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:35.539721 11856 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:35.280084 15177 consensus_queue.cc:1159] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" responder_term: 6 status { last_received { term: 5 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250113 08:59:35.541554 11931 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.563004 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:35.518271 15000 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.027s sys 0.012s
I20250113 08:59:35.548524 12034 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:35.543310 15044 raft_consensus.cc:738] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
W20250113 08:59:35.549803 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 658) took 4517 ms (client timeout 1653 ms). Trace:
W20250113 08:59:35.577412 12719 rpcz_store.cc:269] 0113 08:59:31.032739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.033265 (+ 526us) service_pool.cc:225] Handling call
0113 08:59:35.549548 (+4516283us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54004105d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:35.549698 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4568974208}
I20250113 08:59:35.547151 11962 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:35.563716 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:35.579651 12036 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:35.567710 11931 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.581444 11931 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:35.546793 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:35.569191 11859 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.000s sys 0.055s
I20250113 08:59:35.571038 11856 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:35.528426 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1994) took 4524 ms. Trace:
I20250113 08:59:35.582271 12037 raft_consensus.cc:2463] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
W20250113 08:59:35.582551 11859 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.514s user 0.000s sys 0.055s
I20250113 08:59:35.582428 11886 rpcz_store.cc:276] 0113 08:59:31.003771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.072286 (+ 68515us) service_pool.cc:225] Handling call
0113 08:59:31.117327 (+ 45041us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.335028 (+4217701us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54003118a0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336599990 00007fc33660b300 00007fc33660c371 00007fc336608edf 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc
0113 08:59:35.527993 (+192965us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:35.528025 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:35.528076 (+ 51us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:35.528124 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.528211 (+ 87us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:35.528340 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55050,"raft.queue_time_us":56605,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"spinlock_wait_cycles":3302074240}
I20250113 08:59:35.584067 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:35.584447 12037 raft_consensus.cc:1270] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250113 08:59:35.584697 12033 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.584472 11886 raft_consensus.cc:1270] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20250113 08:59:35.590570 11886 pending_rounds.cc:77] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 5
I20250113 08:59:35.590862 11886 pending_rounds.cc:99] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250113 08:59:35.591141 11886 raft_consensus.cc:2883] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:35.591512 11886 pending_rounds.cc:99] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250113 08:59:35.591742 11886 raft_consensus.cc:2883] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:35.578825 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 429196us
I20250113 08:59:35.574904 14783 raft_consensus.cc:2798] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:35.589712 11931 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:35.592621 12034 raft_consensus.cc:2463] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 8.
I20250113 08:59:35.606655 12034 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:35.607969 12034 raft_consensus.cc:1235] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:35.604588 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1866) took 2556 ms. Trace:
I20250113 08:59:35.609611 11886 rpcz_store.cc:276] 0113 08:59:33.048333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.583925 (+2535592us) service_pool.cc:225] Handling call
0113 08:59:35.584177 (+ 252us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.604252 (+ 20075us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.604511 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:35.611317 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 607) took 5732 ms. Trace:
I20250113 08:59:35.616909 12033 raft_consensus.cc:2463] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:35.635097 15194 raft_consensus.cc:2798] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:35.636267 15194 raft_consensus.cc:695] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.579327 15109 raft_consensus.cc:695] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:35.453413 15172 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.641594 12033 raft_consensus.cc:1270] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250113 08:59:35.613682 14956 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:35.642654 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:35.637753 14736 raft_consensus.cc:2798] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:35.643731 14736 raft_consensus.cc:695] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.644215 12719 raft_consensus.cc:1235] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:35.594695 15058 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.032s sys 0.025s
I20250113 08:59:35.611824 11892 rpcz_store.cc:276] 0113 08:59:29.878937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.098042 (+1219105us) service_pool.cc:225] Handling call
0113 08:59:31.098345 (+ 303us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:31.098591 (+ 246us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:31.230950 (+132359us) log.cc:844] Serialized 14 byte log entry
0113 08:59:31.415677 (+184727us) log.cc:844] Serialized 14 byte log entry
0113 08:59:32.383210 (+967533us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b18000c4720. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d10d8 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:59:32.566894 (+183684us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b18000c4720. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d10d8 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:59:35.610766 (+3043872us) spinlock_profiling.cc:243] Waited 88.3 ms on lock 0x7b18000c4720. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc33629408c 00007fc3336b34ec 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336cdbf5 00007fc3366013d0 00007fc336613666 00007fc3366241d7
0113 08:59:35.610895 (+ 129us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:35.610959 (+ 64us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:35.611025 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.611113 (+ 88us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:35.611250 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":392124,"prepare.queue_time_us":599333,"prepare.run_cpu_time_us":164,"prepare.run_wall_time_us":13593,"raft.queue_time_us":411307,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":15290,"spinlock_wait_cycles":246876672,"thread_start_us":546653,"threads_started":3,"wal-append.queue_time_us":839014,"wal-append.run_cpu_time_us":2023,"wal-append.run_wall_time_us":1256117}
I20250113 08:59:35.650852 14736 consensus_queue.cc:237] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.613607 15073 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:35.719614 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 129995us
W20250113 08:59:35.722190 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 151265us
I20250113 08:59:35.720500 12037 raft_consensus.cc:1235] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20250113 08:59:35.601497 14717 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
W20250113 08:59:35.697285 12718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 655) took 4667 ms (client timeout 1778 ms). Trace:
W20250113 08:59:35.723546 12718 rpcz_store.cc:269] 0113 08:59:31.030227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.030488 (+ 261us) service_pool.cc:225] Handling call
0113 08:59:35.697071 (+4666583us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54007026d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:35.697182 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":617489152}
I20250113 08:59:35.725431 12718 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:35.579188 14729 consensus_queue.cc:237] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.724990 11931 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:35.725939 14953 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:35.641399 15005 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:35.638619 14865 consensus_queue.cc:1035] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20250113 08:59:35.721380 12036 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20250113 08:59:35.774513 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:35.651561 15058 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.032s sys 0.025s
I20250113 08:59:35.650063 14783 raft_consensus.cc:695] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.772230 12719 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:35.775213 12718 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
W20250113 08:59:35.706616 15089 log.cc:926] Time spent T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 3.995s user 0.004s sys 0.028s
I20250113 08:59:35.654099 15046 consensus_queue.cc:1046] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
W20250113 08:59:35.713380 15105 consensus_meta.cc:287] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.813s user 0.013s sys 0.001s
W20250113 08:59:35.777873 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 55232.5us
W20250113 08:59:35.727090 15059 consensus_meta.cc:287] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.817s user 0.011s sys 0.003s
I20250113 08:59:35.787745 11931 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:35.786350 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:35.729737 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 984885us
I20250113 08:59:35.727996 12034 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20250113 08:59:35.789567 12006 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:35.723047 11856 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:35.776664 11857 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.018s sys 0.011s
I20250113 08:59:35.688283 14950 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:35.660372 15195 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:35.678910 15108 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:35.777359 14953 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:35.776050 14976 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.001s sys 0.018s
I20250113 08:59:35.777746 14717 raft_consensus.cc:695] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.803057 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:35.806751 12035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1745) took 4677 ms (client timeout 2348 ms). Trace:
I20250113 08:59:35.807334 12719 raft_consensus.cc:1235] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:35.807163 12035 rpcz_store.cc:269] 0113 08:59:31.124049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.127188 (+ 3139us) service_pool.cc:225] Handling call
0113 08:59:35.799421 (+4672233us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b540002d0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:35.801407 (+ 1986us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36885888}
W20250113 08:59:35.807811 11857 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.019s sys 0.011s
I20250113 08:59:35.787545 14937 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:35.815516 15035 consensus_queue.cc:1046] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20250113 08:59:35.676640 15109 consensus_queue.cc:237] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:35.803834 14976 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s user 0.001s sys 0.019s
I20250113 08:59:35.727391 15129 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:35.810848 14950 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250113 08:59:35.819975 14954 consensus_meta.cc:287] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.905s user 0.013s sys 0.000s
I20250113 08:59:35.721407 15194 consensus_queue.cc:237] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.835212 11889 raft_consensus.cc:1235] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 4. Current term is 5. Ops: []
W20250113 08:59:35.835177 11893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1995) took 4831 ms (client timeout 1765 ms). Trace:
I20250113 08:59:35.803537 15058 raft_consensus.cc:2798] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:59:35.640307 15044 consensus_queue.cc:260] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.796556 15105 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:35.846683 11893 rpcz_store.cc:269] 0113 08:59:31.004107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.027195 (+ 23088us) service_pool.cc:225] Handling call
0113 08:59:35.834887 (+4807692us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b540038e854. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:35.835067 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4797370112}
I20250113 08:59:35.850847 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:35.838122 15040 consensus_meta.cc:287] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.727s user 0.010s sys 0.003s
I20250113 08:59:35.851678 11893 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 11.
I20250113 08:59:35.851886 15197 raft_consensus.cc:2798] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:35.855520 15197 raft_consensus.cc:695] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.832572 14937 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:35.859126 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:35.854441 12723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 659) took 3400 ms (client timeout 2348 ms). Trace:
W20250113 08:59:35.860505 12723 rpcz_store.cc:269] 0113 08:59:32.453741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.453904 (+ 163us) service_pool.cc:225] Handling call
0113 08:59:35.852396 (+3398492us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b5400751554. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:35.854354 (+ 1958us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4068449536}
I20250113 08:59:35.797596 15059 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.863354 12723 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250113 08:59:35.840379 14717 consensus_queue.cc:237] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.865797 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:35.872133 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:35.805192 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 82227.1us
I20250113 08:59:35.872992 11893 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:35.873706 11856 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:35.874769 14950 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:35.875224 14950 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:35.875656 14950 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:35.882625 11964 raft_consensus.cc:1235] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: []
I20250113 08:59:35.859649 15058 raft_consensus.cc:695] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:35.877041 11856 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:35.902801 11856 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:35.847234 14954 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:35.837225 14783 consensus_queue.cc:237] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:35.959635 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 56946.1us
I20250113 08:59:35.957329 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:35.962378 11934 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:35.911185 15077 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 4.520s user 0.015s sys 0.048s
W20250113 08:59:35.933317 14940 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s user 0.015s sys 0.050s
I20250113 08:59:35.856807 15197 consensus_queue.cc:237] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:35.958653 14990 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.030s sys 0.060s
I20250113 08:59:36.007310 12720 raft_consensus.cc:2388] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:35.860361 15040 raft_consensus.cc:513] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:35.954219 15196 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250113 08:59:36.015653 12685 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.008s sys 0.004s
W20250113 08:59:36.043237 12685 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.639s user 0.009s sys 0.004s
W20250113 08:59:36.029714 15116 log.cc:926] Time spent T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 3.229s user 0.012s sys 0.022s
I20250113 08:59:36.016547 15195 consensus_queue.cc:1046] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20250113 08:59:35.898438 15031 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.116s
I20250113 08:59:36.038647 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1859) took 3089 ms. Trace:
W20250113 08:59:36.059994 12721 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.000s sys 0.029s
W20250113 08:59:36.060464 12721 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s user 0.000s sys 0.029s
I20250113 08:59:36.039242 14953 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:36.015905 14940 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s user 0.016s sys 0.051s
W20250113 08:59:36.019984 14990 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s user 0.030s sys 0.060s
W20250113 08:59:36.040390 15077 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 4.632s user 0.016s sys 0.049s
W20250113 08:59:36.057510 12040 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.023s sys 0.077s
W20250113 08:59:36.082155 12040 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s user 0.023s sys 0.077s
I20250113 08:59:36.058518 11890 rpcz_store.cc:276] 0113 08:59:32.948987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.082675 (+2133688us) service_pool.cc:225] Handling call
0113 08:59:35.111359 (+ 28684us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:35.111732 (+ 373us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:35.111768 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:35.125068 (+ 13300us) log.cc:844] Serialized 26 byte log entry
0113 08:59:35.575934 (+450866us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:35.575988 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:35.576098 (+ 110us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:36.038377 (+462279us) raft_consensus.cc:1680] finished
0113 08:59:36.038401 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:36.038564 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":68895,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":100,"thread_start_us":435893,"threads_started":1,"wal-append.queue_time_us":778514}
I20250113 08:59:36.022311 15172 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:36.083670 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:36.089738 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 312923us
W20250113 08:59:36.085711 12042 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.000s sys 0.017s
I20250113 08:59:36.046036 15196 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:36.093868 12042 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.000s sys 0.017s
I20250113 08:59:36.014199 15199 consensus_queue.cc:1035] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.137s
I20250113 08:59:36.120633 12685 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:36.498777 14954 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:36.214434 14937 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:59:36.499370 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 378855us
I20250113 08:59:36.499522 14964 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250113 08:59:36.500202 14964 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:36.101450 15195 consensus_queue.cc:1225] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:36.500905 14964 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 11 FOLLOWER]: Advancing to term 12
I20250113 08:59:36.137607 15204 consensus_queue.cc:1046] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.415s
I20250113 08:59:36.072182 15058 consensus_queue.cc:237] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:36.502606 15204 raft_consensus.cc:3049] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:36.503050 15204 raft_consensus.cc:738] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:36.157145 11893 raft_consensus.cc:2463] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
W20250113 08:59:36.379419 11859 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.007s sys 0.020s
W20250113 08:59:36.506819 11859 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.724s user 0.009s sys 0.025s
W20250113 08:59:36.498112 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 361158us
I20250113 08:59:36.115463 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:36.507239 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 725179us
I20250113 08:59:36.499902 14937 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:36.524874 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:36.241109 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 669) took 2897 ms (client timeout 1806 ms). Trace:
W20250113 08:59:36.543344 12717 rpcz_store.cc:269] 0113 08:59:33.343951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.346289 (+ 2338us) service_pool.cc:225] Handling call
0113 08:59:36.240833 (+2894544us) spinlock_profiling.cc:243] Waited 67 ms on lock 0x7b54007503d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:36.241007 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154157312}
W20250113 08:59:36.549510 11932 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.010s sys 0.023s
W20250113 08:59:36.550051 11932 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.899s user 0.010s sys 0.023s
W20250113 08:59:36.305202 12686 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.954s user 0.034s sys 0.095s
I20250113 08:59:36.110723 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 822) took 1729 ms. Trace:
W20250113 08:59:36.561764 12686 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.211s user 0.040s sys 0.101s
I20250113 08:59:36.563341 12721 rpcz_store.cc:276] 0113 08:59:34.381311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.398084 (+1016773us) service_pool.cc:225] Handling call
0113 08:59:35.402381 (+ 4297us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.402596 (+ 215us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:35.421601 (+ 19005us) log.cc:844] Serialized 14 byte log entry
0113 08:59:36.109944 (+688343us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:36.110324 (+ 380us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:36.110391 (+ 67us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:36.110484 (+ 93us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:36.110648 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57376,"prepare.queue_time_us":25281,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":56150,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":28781,"thread_start_us":638954,"threads_started":1,"wal-append.queue_time_us":1017507}
W20250113 08:59:36.181929 15201 log.cc:926] Time spent T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.316s user 0.000s sys 0.002s
W20250113 08:59:36.102289 15066 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.006s sys 0.009s
I20250113 08:59:36.101958 14953 consensus_queue.cc:1035] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.491s
W20250113 08:59:36.393973 15159 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.066s user 0.006s sys 0.027s
W20250113 08:59:36.609728 15159 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s user 0.006s sys 0.032s
W20250113 08:59:36.258390 15060 consensus_meta.cc:287] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.886s user 0.012s sys 0.002s
I20250113 08:59:36.610742 15060 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:36.078477 14795 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.044s sys 0.054s
W20250113 08:59:36.613507 14795 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.466s user 0.045s sys 0.076s
I20250113 08:59:36.438704 15168 raft_consensus.cc:3049] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:36.614584 15168 raft_consensus.cc:738] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:36.473548 15019 consensus_meta.cc:287] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 1.346s user 0.009s sys 0.006s
W20250113 08:59:36.446309 15202 log.cc:926] Time spent T 39676b99e4434f11b6017bb867fb7471 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.281s user 0.000s sys 0.000s
I20250113 08:59:36.504128 15204 consensus_queue.cc:260] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:36.616575 15204 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:36.114861 15196 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:36.233958 15059 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:36.168113 15040 leader_election.cc:290] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:36.215147 15105 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:36.182195 15127 raft_consensus.cc:3049] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
W20250113 08:59:36.600855 15066 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.058s user 0.006s sys 0.030s
I20250113 08:59:36.413280 15200 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:36.632944 15041 consensus_meta.cc:287] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.075s user 0.013s sys 0.000s
W20250113 08:59:36.663122 15109 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.021s sys 0.008s
W20250113 08:59:36.680667 15109 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s user 0.021s sys 0.008s
I20250113 08:59:36.662534 15127 raft_consensus.cc:738] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:36.673322 15060 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:36.776495 15127 consensus_queue.cc:260] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:36.784045 15127 raft_consensus.cc:3054] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:36.881839 11886 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.239s user 0.012s sys 0.019s
W20250113 08:59:36.882854 11886 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.240s user 0.012s sys 0.020s
I20250113 08:59:36.883706 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:36.889546 11931 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.100s user 0.006s sys 0.007s
I20250113 08:59:36.889770 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1892) took 1243 ms. Trace:
W20250113 08:59:36.890090 11931 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.100s user 0.006s sys 0.007s
I20250113 08:59:36.890306 11890 rpcz_store.cc:276] 0113 08:59:35.646279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:36.525775 (+879496us) service_pool.cc:225] Handling call
0113 08:59:36.889596 (+363821us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:36.895308 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1763) took 1545 ms. Trace:
W20250113 08:59:36.897748 12718 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.098s user 0.008s sys 0.037s
W20250113 08:59:36.898412 12718 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.098s user 0.008s sys 0.037s
W20250113 08:59:36.898484 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.935s user 0.009s sys 0.005s
W20250113 08:59:36.899225 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.936s user 0.009s sys 0.005s
W20250113 08:59:36.899580 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 1.54939e+06us
W20250113 08:59:36.900341 12719 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.070s user 0.000s sys 0.011s
W20250113 08:59:36.901003 12719 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.071s user 0.000s sys 0.011s
W20250113 08:59:36.902577 12724 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.523s user 0.019s sys 0.027s
W20250113 08:59:36.903159 12724 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.523s user 0.019s sys 0.027s
W20250113 08:59:36.904639 15005 consensus_meta.cc:287] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.127s user 0.011s sys 0.010s
I20250113 08:59:36.904862 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 887) took 2938 ms. Trace:
I20250113 08:59:36.561936 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:36.660367 14736 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.025s sys 0.030s
W20250113 08:59:36.919781 15046 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.113s user 0.008s sys 0.043s
W20250113 08:59:36.920603 15046 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.121s user 0.009s sys 0.043s
W20250113 08:59:36.919986 14736 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.146s user 0.026s sys 0.030s
I20250113 08:59:36.905695 12724 rpcz_store.cc:276] 0113 08:59:33.966346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.338021 (+1371675us) service_pool.cc:225] Handling call
0113 08:59:35.379312 (+ 41291us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:36.904058 (+1524746us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:36.904127 (+ 69us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:36.904230 (+ 103us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:36.904288 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:36.904434 (+ 146us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:36.904741 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":151568,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":1523804,"threads_started":1}
I20250113 08:59:36.663172 15041 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:36.926148 14736 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:36.926852 12686 connection.cc:232] RPC call timeout handler was delayed by -1.4473658769999247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20250113 08:59:36.927467 14736 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:36.895959 12042 rpcz_store.cc:276] 0113 08:59:35.349996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.350126 (+ 130us) service_pool.cc:225] Handling call
0113 08:59:35.350436 (+ 310us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:35.380593 (+ 30157us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:35.380621 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:35.388492 (+ 7871us) log.cc:844] Serialized 26 byte log entry
0113 08:59:36.498718 (+1110226us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:36.498779 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:36.498884 (+ 105us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:36.894475 (+395591us) raft_consensus.cc:1680] finished
0113 08:59:36.894538 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:36.894984 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29758,"raft.queue_time_us":32860,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":1756,"thread_start_us":1070044,"threads_started":1,"wal-append.queue_time_us":1389472}
I20250113 08:59:36.151398 14940 consensus_queue.cc:1046] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.513s
I20250113 08:59:36.931690 11893 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:36.933846 11893 raft_consensus.cc:1235] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:36.933789 12685 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:36.935546 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 973222us
I20250113 08:59:36.905314 15005 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:36.939036 14736 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:36.895961 11932 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:36.946539 15040 consensus_queue.cc:1035] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:36.950255 15059 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:36.948793 15040 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:36.951305 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1893) took 1221 ms. Trace:
I20250113 08:59:36.997700 11893 rpcz_store.cc:276] 0113 08:59:35.722053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:36.932892 (+1210839us) service_pool.cc:225] Handling call
0113 08:59:36.933556 (+ 664us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:36.934649 (+ 1093us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:36.943807 (+ 9158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:37.033694 15196 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:37.077783 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1762) took 1731 ms. Trace:
W20250113 08:59:37.086493 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 158542us
I20250113 08:59:37.037554 14736 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250113 08:59:37.093323 14950 consensus_meta.cc:287] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 1.043s user 0.009s sys 0.007s
I20250113 08:59:37.093626 14937 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:37.094707 15228 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:37.096081 11891 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 617) took 6074 ms (client timeout 1921 ms). Trace:
W20250113 08:59:37.096720 15123 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.896s user 0.017s sys 0.015s
W20250113 08:59:37.102262 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 525) took 3855 ms (client timeout 1708 ms). Trace:
I20250113 08:59:37.087589 15040 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.103085 15196 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:37.103607 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:37.103910 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1857) took 4155 ms. Trace:
I20250113 08:59:37.093387 14954 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250113 08:59:37.109215 11891 rpcz_store.cc:269] 0113 08:59:31.021345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.065870 (+ 44525us) service_pool.cc:225] Handling call
0113 08:59:37.095522 (+6029652us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b540034d0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.095862 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4453127040}
W20250113 08:59:37.112107 11963 rpcz_store.cc:269] 0113 08:59:33.242316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.261700 (+ 19384us) service_pool.cc:225] Handling call
0113 08:59:37.096942 (+3835242us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540006ead4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.097219 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3025276160}
I20250113 08:59:37.112742 11889 rpcz_store.cc:276] 0113 08:59:32.948328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.965855 (+2017527us) service_pool.cc:225] Handling call
0113 08:59:35.835036 (+869181us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.846603 (+ 11567us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.103395 (+1256792us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b540038e850. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.103709 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1997816704}
W20250113 08:59:37.121102 15123 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.934s user 0.018s sys 0.015s
I20250113 08:59:37.115849 14954 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:37.125311 14783 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.007s sys 0.021s
W20250113 08:59:37.145226 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2245) took 6135 ms (client timeout 1830 ms). Trace:
I20250113 08:59:37.145748 11966 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:37.083175 15059 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:37.146507 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:37.153458 15208 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.022s sys 0.055s
I20250113 08:59:37.171810 15215 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:37.088006 12040 rpcz_store.cc:276] 0113 08:59:35.346384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.346859 (+ 475us) service_pool.cc:225] Handling call
0113 08:59:35.347258 (+ 399us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:35.379965 (+ 32707us) raft_consensus.cc:1529] Early marking committed up to index 5
0113 08:59:35.379998 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:35.380457 (+ 459us) log.cc:844] Serialized 26 byte log entry
0113 08:59:36.082647 (+702190us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:36.082704 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:36.082825 (+ 121us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:37.068707 (+985882us) raft_consensus.cc:1680] finished
0113 08:59:37.071635 (+ 2928us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:37.077578 (+ 5943us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":266131,"raft.queue_time_us":34122,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4881,"thread_start_us":675479,"threads_started":1,"wal-append.queue_time_us":1374584}
I20250113 08:59:37.192149 15019 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.150871 11961 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:37.151449 15194 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.018s user 0.019s sys 0.039s
W20250113 08:59:37.151633 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 536) took 1958 ms (client timeout 1985 ms). Trace:
W20250113 08:59:37.177702 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 282284us
I20250113 08:59:37.193439 15019 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:37.193885 11962 rpcz_store.cc:269] 0113 08:59:35.192937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.238440 (+ 45503us) service_pool.cc:225] Handling call
0113 08:59:37.151212 (+1912772us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b540015d954. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.151447 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":636493696}
I20250113 08:59:37.195554 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:37.196383 11962 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:37.146425 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
W20250113 08:59:37.193583 15194 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.150s user 0.019s sys 0.039s
I20250113 08:59:37.170900 15220 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:59:37.197077 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 530) took 2067 ms (client timeout 2111 ms). Trace:
W20250113 08:59:37.199529 11962 rpcz_store.cc:269] 0113 08:59:35.129686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.195061 (+2065375us) service_pool.cc:225] Handling call
0113 08:59:37.196979 (+ 1918us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:37.152416 12006 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.334s user 0.003s sys 0.015s
I20250113 08:59:37.152942 15041 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:37.153800 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 638) took 1342 ms. Trace:
W20250113 08:59:37.191761 15208 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.150s user 0.023s sys 0.056s
W20250113 08:59:37.202495 11960 rpcz_store.cc:269] 0113 08:59:30.968697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:30.968855 (+ 158us) service_pool.cc:225] Handling call
0113 08:59:37.103965 (+6135110us) spinlock_profiling.cc:243] Waited 3.55 s on lock 0x7b54001ae5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.104233 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8157388032}
I20250113 08:59:37.152344 15231 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:37.132840 14717 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.006s sys 0.004s
I20250113 08:59:37.220594 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:37.173013 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:37.188288 15040 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:37.221803 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.026s user 0.017s sys 0.045s
W20250113 08:59:37.153065 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 148762us
I20250113 08:59:37.176999 15228 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:37.176398 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2291) took 4831 ms (client timeout 4529 ms). Trace:
I20250113 08:59:37.173766 12721 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
W20250113 08:59:37.228519 12006 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.434s user 0.004s sys 0.016s
I20250113 08:59:37.175220 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2265) took 4205 ms. Trace:
I20250113 08:59:37.223662 11891 rpcz_store.cc:276] 0113 08:59:35.811376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.152299 (+1340923us) service_pool.cc:225] Handling call
0113 08:59:37.152862 (+ 563us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:37.153403 (+ 541us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:37.153444 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:37.153486 (+ 42us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:37.153532 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.153599 (+ 67us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:37.153732 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":32812,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105}
I20250113 08:59:37.153934 14954 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:37.196794 11932 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:37.199853 15220 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:37.260272 14783 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.011s sys 0.021s
W20250113 08:59:37.276372 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.204s user 0.018s sys 0.045s
I20250113 08:59:37.221909 15215 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.277263 14951 raft_consensus.cc:3054] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:37.280058 15059 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:37.182008 14455 raft_consensus.cc:3049] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:37.282094 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:37.283535 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 831) took 1557 ms. Trace:
W20250113 08:59:37.283584 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37184 (request call id 619) took 3765 ms (client timeout 2037 ms). Trace:
I20250113 08:59:37.275863 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 633) took 3448 ms. Trace:
I20250113 08:59:37.121979 14950 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:37.283396 15197 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.029s sys 0.033s
W20250113 08:59:37.271677 11968 rpcz_store.cc:269] 0113 08:59:32.345230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.242297 (+897067us) service_pool.cc:225] Handling call
0113 08:59:37.175949 (+3933652us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400050ad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.176186 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3202951936}
W20250113 08:59:37.275379 14717 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s user 0.006s sys 0.010s
I20250113 08:59:37.272182 11964 rpcz_store.cc:276] 0113 08:59:32.970046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.747285 (+1777239us) service_pool.cc:225] Handling call
0113 08:59:34.747673 (+ 388us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.898326 (+1150653us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.174661 (+1276335us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54002a3354. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:37.175019 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2609787648}
W20250113 08:59:37.290860 12686 connection.cc:232] RPC call timeout handler was delayed by -1.3422731140000224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1998s for a response
W20250113 08:59:37.299544 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2310) took 2099 ms (client timeout 2132 ms). Trace:
I20250113 08:59:37.292761 11963 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
W20250113 08:59:37.300840 11960 rpcz_store.cc:269] 0113 08:59:35.196429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.281659 (+2085230us) service_pool.cc:225] Handling call
0113 08:59:37.295577 (+ 13918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:37.295354 12719 rpcz_store.cc:276] 0113 08:59:35.726472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.825666 (+ 99194us) service_pool.cc:225] Handling call
0113 08:59:35.826341 (+ 675us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:35.828196 (+ 1855us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:35.828235 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:35.829642 (+ 1407us) log.cc:844] Serialized 572 byte log entry
0113 08:59:36.901897 (+1072255us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:36.902018 (+ 121us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:36.902186 (+ 168us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:37.283216 (+381030us) raft_consensus.cc:1680] finished
0113 08:59:37.283237 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:37.283438 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1071472,"threads_started":1,"wal-append.queue_time_us":1450932}
W20250113 08:59:37.298275 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 101704us
W20250113 08:59:37.301119 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 539) took 2107 ms (client timeout 2037 ms). Trace:
W20250113 08:59:37.303216 11963 rpcz_store.cc:269] 0113 08:59:35.194111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.145845 (+1951734us) service_pool.cc:225] Handling call
0113 08:59:37.301000 (+155155us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:37.296883 12041 rpcz_store.cc:269] 0113 08:59:33.518157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.837629 (+319472us) service_pool.cc:225] Handling call
0113 08:59:37.283074 (+3445445us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b54000cdad4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.283311 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1083362560}
W20250113 08:59:37.302212 12686 connection.cc:232] RPC call timeout handler was delayed by -1.3221358680000321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.213s for a response
W20250113 08:59:37.298099 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.22224e+06us
I20250113 08:59:37.301280 11962 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
W20250113 08:59:37.304885 12686 connection.cc:232] RPC call timeout handler was delayed by -1.181528070000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
I20250113 08:59:37.308066 15019 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:37.309154 14845 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.309955 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:37.310036 14845 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.295066 14455 raft_consensus.cc:738] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:37.312817 14455 consensus_queue.cc:260] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.314144 14455 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:37.316713 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:37.318033 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:37.386232 11932 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:37.411059 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:37.439006 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:37.340420 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 673) took 3611 ms. Trace:
I20250113 08:59:37.296996 11890 rpcz_store.cc:276] 0113 08:59:33.820992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:36.941298 (+3120306us) service_pool.cc:225] Handling call
0113 08:59:36.946052 (+ 4754us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:36.946621 (+ 569us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:36.947365 (+ 744us) log.cc:844] Serialized 14 byte log entry
0113 08:59:37.268464 (+321099us) log.cc:844] Serialized 14 byte log entry
0113 08:59:37.268711 (+ 247us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:37.268833 (+ 122us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:37.268902 (+ 69us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.268974 (+ 72us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:37.269148 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":62198,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":2018,"raft.queue_time_us":62838,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"spinlock_wait_cycles":6608640,"thread_start_us":289613,"threads_started":2}
I20250113 08:59:37.439710 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:37.325197 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 535) took 2113 ms (client timeout 2130 ms). Trace:
I20250113 08:59:37.387218 11885 raft_consensus.cc:1270] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20250113 08:59:37.432898 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:37.391050 11960 raft_consensus.cc:2371] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 8. Re-sending same reply.
W20250113 08:59:37.301772 15197 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s user 0.030s sys 0.034s
W20250113 08:59:37.410605 15237 log.cc:926] Time spent T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
W20250113 08:59:37.445689 11962 rpcz_store.cc:269] 0113 08:59:35.192391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.200689 (+2008298us) service_pool.cc:225] Handling call
0113 08:59:37.305850 (+105161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:37.446409 11885 pending_rounds.cc:77] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 1
W20250113 08:59:37.319938 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 1.53034e+06us
W20250113 08:59:37.340890 15127 consensus_meta.cc:287] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.556s user 0.000s sys 0.018s
I20250113 08:59:37.345662 15123 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:37.346378 14964 consensus_meta.cc:287] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.844s user 0.006s sys 0.016s
I20250113 08:59:37.352846 15059 raft_consensus.cc:2798] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:37.441671 12726 rpcz_store.cc:276] 0113 08:59:33.728825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.100856 (+1372031us) service_pool.cc:225] Handling call
0113 08:59:35.101435 (+ 579us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.139208 (+ 37773us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.339854 (+2200646us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b540031ea54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:37.340216 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78745344}
I20250113 08:59:37.425037 14994 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:37.423556 14950 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:37.489657 12719 raft_consensus.cc:2388] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:37.488842 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 641) took 1437 ms. Trace:
I20250113 08:59:37.490906 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:37.338466 15231 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.492651 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:37.461633 14790 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.461939 14783 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.469909 14956 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:37.470098 15224 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.471304 15053 raft_consensus.cc:3049] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:37.470913 15123 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:37.472391 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 104759us
I20250113 08:59:37.472625 15215 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:37.473049 15040 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:37.474416 15236 log.cc:926] Time spent T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20250113 08:59:37.474232 14964 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.475440 15220 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:37.475662 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 538) took 2281 ms (client timeout 2018 ms). Trace:
W20250113 08:59:37.477540 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 542) took 2282 ms (client timeout 2223 ms). Trace:
I20250113 08:59:37.483878 15061 raft_consensus.cc:3049] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:37.487433 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:37.490970 15065 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:37.495215 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 69159.3us
I20250113 08:59:37.491079 11891 rpcz_store.cc:276] 0113 08:59:36.051237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.486135 (+1434898us) service_pool.cc:225] Handling call
0113 08:59:37.486989 (+ 854us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:37.487725 (+ 736us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:37.487781 (+ 56us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:37.487876 (+ 95us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:37.488155 (+ 279us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.488245 (+ 90us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:37.488486 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:37.455924 11885 pending_rounds.cc:99] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250113 08:59:37.505124 12723 raft_consensus.cc:2388] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:37.494042 15068 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.517688 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:37.518666 12719 raft_consensus.cc:2405] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:37.519320 12721 raft_consensus.cc:2405] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:37.494187 15228 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:37.522387 15238 log.cc:926] Time spent T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.171s user 0.001s sys 0.001s
I20250113 08:59:37.522262 15224 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:37.520809 11934 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:37.535563 12039 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:37.514458 15053 raft_consensus.cc:738] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:37.537055 12686 connection.cc:232] RPC call timeout handler was delayed by -1.0693503950000149s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2037s for a response
W20250113 08:59:37.540599 11963 rpcz_store.cc:269] 0113 08:59:35.195172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.392145 (+2196973us) service_pool.cc:225] Handling call
0113 08:59:37.477430 (+ 85285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:37.543771 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:37.544997 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 69028.7us
I20250113 08:59:37.543764 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:37.540499 11967 rpcz_store.cc:269] 0113 08:59:35.193751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.385067 (+191316us) service_pool.cc:225] Handling call
0113 08:59:37.475266 (+2090199us) spinlock_profiling.cc:243] Waited 2.08 s on lock 0x7b540039fb54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:37.475458 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4779521792}
I20250113 08:59:37.535306 15041 raft_consensus.cc:491] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.544453 11885 raft_consensus.cc:2883] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:37.524258 15059 raft_consensus.cc:695] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:37.522419 14790 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.542683 15194 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.649580 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:37.655139 12036 raft_consensus.cc:1235] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
W20250113 08:59:37.812604 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 263722us
I20250113 08:59:38.102470 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 682) took 2509 ms. Trace:
I20250113 08:59:37.656273 12035 raft_consensus.cc:1235] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:38.110451 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 688) took 1171 ms. Trace:
I20250113 08:59:38.122752 11885 pending_rounds.cc:99] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250113 08:59:37.656993 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:37.691368 11890 raft_consensus.cc:1270] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250113 08:59:38.161306 11885 raft_consensus.cc:2883] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:37.752542 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:37.547816 15031 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.161046 11886 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 629) took 4990 ms. Trace:
I20250113 08:59:37.541954 15061 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:37.756016 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:37.582451 15244 log.cc:926] Time spent T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20250113 08:59:37.968236 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 277393us
I20250113 08:59:37.697435 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:38.067286 15104 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.539342 14950 consensus_queue.cc:1046] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.587s
I20250113 08:59:37.657914 12686 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:37.727442 14964 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 12 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:37.645776 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:38.122653 15065 consensus_meta.cc:287] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.540s user 0.010s sys 0.005s
W20250113 08:59:38.168967 12039 consensus_meta.cc:287] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.511s user 0.006s sys 0.006s
I20250113 08:59:37.730276 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 622) took 5187 ms. Trace:
W20250113 08:59:38.212386 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 662207us
I20250113 08:59:37.664248 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.067037 14944 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:38.193534 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2312) took 2363 ms (client timeout 2007 ms). Trace:
I20250113 08:59:37.553171 15228 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:38.160980 12724 rpcz_store.cc:276] 0113 08:59:36.939149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:36.939347 (+ 198us) service_pool.cc:225] Handling call
0113 08:59:36.940043 (+ 696us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:36.940719 (+ 676us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:36.940751 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:36.941923 (+ 1172us) log.cc:844] Serialized 52 byte log entry
0113 08:59:37.117153 (+175230us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:37.117208 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.117324 (+ 116us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:37.585400 (+468076us) raft_consensus.cc:1680] finished
0113 08:59:37.585423 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:38.110378 (+524955us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":64710,"raft.run_cpu_time_us":549,"raft.run_wall_time_us":75329,"thread_start_us":174703,"threads_started":1,"wal-append.queue_time_us":640306}
I20250113 08:59:38.117887 15157 raft_consensus.cc:491] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.726817 15171 raft_consensus.cc:3049] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:37.658330 15255 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.047991 15187 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:38.162591 12718 rpcz_store.cc:276] 0113 08:59:35.593080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.781914 (+188834us) service_pool.cc:225] Handling call
0113 08:59:35.782275 (+ 361us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:35.793627 (+ 11352us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:35.793673 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:35.798592 (+ 4919us) log.cc:844] Serialized 52 byte log entry
0113 08:59:36.899563 (+1100971us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:36.899735 (+ 172us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:36.899967 (+ 232us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:38.102136 (+1202169us) raft_consensus.cc:1680] finished
0113 08:59:38.102166 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:38.102383 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10866,"raft.queue_time_us":13501,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4120,"spinlock_wait_cycles":1765632,"thread_start_us":1099060,"threads_started":1,"wal-append.queue_time_us":2282393}
I20250113 08:59:38.048970 15224 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:38.100172 15179 raft_consensus.cc:3049] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:37.584287 15251 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
W20250113 08:59:38.219925 11967 rpcz_store.cc:269] 0113 08:59:35.800516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.133705 (+2333189us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.163893 (+ 30188us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:37.659883 15146 raft_consensus.cc:3049] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:38.029703 15261 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.216972 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:37.707074 15058 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:38.216742 15247 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:37.675034 15231 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:37.552688 15215 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:38.227595 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 421943us
I20250113 08:59:37.674367 15102 raft_consensus.cc:2798] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:38.252514 15199 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.215183 14963 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:37.731056 15123 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:37.661914 15068 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:37.554246 14956 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.231597 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.232075 15091 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.263628 15053 consensus_queue.cc:260] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.218827 12039 raft_consensus.cc:2463] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:38.265061 15053 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:38.265636 12040 raft_consensus.cc:1235] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250113 08:59:38.265929 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 643) took 1385 ms. Trace:
I20250113 08:59:38.265062 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1774) took 1161 ms. Trace:
I20250113 08:59:38.267032 12039 rpcz_store.cc:276] 0113 08:59:37.103635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.128185 (+ 24550us) service_pool.cc:225] Handling call
0113 08:59:38.264966 (+1136781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:38.267903 15061 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:38.268846 15040 consensus_meta.cc:287] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.727s user 0.010s sys 0.002s
I20250113 08:59:38.268709 12039 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250113 08:59:38.266352 11891 rpcz_store.cc:276] 0113 08:59:36.880788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.548295 (+667507us) service_pool.cc:225] Handling call
0113 08:59:38.119212 (+570917us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:38.236419 (+117207us) raft_consensus.cc:1529] Early marking committed up to index 10
0113 08:59:38.243986 (+ 7567us) log.cc:844] Serialized 14 byte log entry
0113 08:59:38.265526 (+ 21540us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:38.265591 (+ 65us) raft_consensus.cc:1640] Marking committed up to 10
0113 08:59:38.265645 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.265719 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:38.265846 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":850,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":73,"thread_start_us":15822,"threads_started":1}
I20250113 08:59:38.217762 11884 rpcz_store.cc:276] 0113 08:59:32.540466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:32.553585 (+ 13119us) service_pool.cc:225] Handling call
0113 08:59:32.590591 (+ 37006us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:32.908159 (+317568us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:32.908190 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:32.908688 (+ 498us) log.cc:844] Serialized 26 byte log entry
0113 08:59:33.454149 (+545461us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:33.454204 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.727690 (+4273486us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54000a3f54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:37.727764 (+ 74us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:37.727892 (+ 128us) raft_consensus.cc:1680] finished
0113 08:59:37.727902 (+ 10us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:37.728199 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3207,"raft.queue_time_us":13687,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":5108,"spinlock_wait_cycles":729416064,"thread_start_us":479132,"threads_started":1,"wal-append.queue_time_us":590618,"wal-append.run_cpu_time_us":2008,"wal-append.run_wall_time_us":1113666}
I20250113 08:59:38.268777 15157 raft_consensus.cc:513] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.270819 11884 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [term 8 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20250113 08:59:38.271553 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 644) took 1073 ms. Trace:
I20250113 08:59:38.272822 15157 leader_election.cc:290] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.230018 15254 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:38.230566 15031 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.222934 15228 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:38.272053 11884 rpcz_store.cc:276] 0113 08:59:37.198609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.270085 (+1071476us) service_pool.cc:225] Handling call
0113 08:59:38.270512 (+ 427us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:38.271283 (+ 771us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.271467 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:38.273485 15254 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:38.273725 15228 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:38.274030 15254 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.275053 15031 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.260603 15104 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.239009 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:38.276676 12036 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:38.162231 11886 rpcz_store.cc:276] 0113 08:59:33.170840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.610558 (+2439718us) service_pool.cc:225] Handling call
0113 08:59:35.640641 (+ 30083us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:35.640868 (+ 227us) raft_consensus.cc:1529] Early marking committed up to index 8
0113 08:59:36.883803 (+1242935us) log.cc:844] Serialized 14 byte log entry
0113 08:59:37.525431 (+641628us) log.cc:844] Serialized 14 byte log entry
0113 08:59:37.537736 (+ 12305us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:37.543336 (+ 5600us) raft_consensus.cc:1640] Marking committed up to 8
0113 08:59:37.550949 (+ 7613us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:37.572755 (+ 21806us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:38.160983 (+588228us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1916191,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":63959,"raft.run_cpu_time_us":437,"raft.run_wall_time_us":6497,"thread_start_us":1720994,"threads_started":4,"wal-append.queue_time_us":466307}
I20250113 08:59:37.584620 15196 consensus_queue.cc:1046] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.645s
I20250113 08:59:38.223929 15171 raft_consensus.cc:738] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:38.244146 12006 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:38.228366 15255 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.244799 14790 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.246748 12035 raft_consensus.cc:1235] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:38.246846 15251 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.247907 15065 raft_consensus.cc:3049] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:38.255556 11890 raft_consensus.cc:3049] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:38.294986 11890 raft_consensus.cc:738] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:37.553748 14783 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:59:38.257942 11892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1889) took 2589 ms (client timeout 1851 ms). Trace:
I20250113 08:59:38.255344 15199 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:38.258337 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 600583us
I20250113 08:59:38.258452 15187 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.230782 15179 raft_consensus.cc:738] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:38.258154 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 664) took 5164 ms. Trace:
I20250113 08:59:38.242622 12041 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9.
I20250113 08:59:38.263473 15091 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.235297 12033 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.269264 15061 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:37.724085 14954 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.231182 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:38.302204 11892 rpcz_store.cc:269] 0113 08:59:35.644651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.232155 (+2587504us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.232324 (+ 169us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:38.238117 15041 raft_consensus.cc:513] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:38.277129 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2314) took 2140 ms (client timeout 1736 ms). Trace:
W20250113 08:59:38.299679 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 83899.9us
I20250113 08:59:38.217432 15194 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.314638 15179 consensus_queue.cc:260] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.241989 15220 consensus_queue.cc:1046] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
W20250113 08:59:38.317260 11967 rpcz_store.cc:269] 0113 08:59:36.136467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.261359 (+2124892us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.277057 (+ 15698us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:38.296350 15215 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.230087 14944 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.319594 11965 raft_consensus.cc:1235] T 87a2e0ccdce34284b13fa0924326b262 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:38.269348 15040 raft_consensus.cc:2743] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:38.337839 15040 raft_consensus.cc:3049] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Stepping down as leader of term 4
I20250113 08:59:38.338289 15040 raft_consensus.cc:738] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:38.339295 15040 consensus_queue.cc:260] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:38.340734 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 540) took 3146 ms (client timeout 2689 ms). Trace:
I20250113 08:59:38.341974 15157 consensus_queue.cc:1046] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250113 08:59:38.342936 15157 consensus_queue.cc:1225] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250113 08:59:38.344450 11856 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:38.319641 14956 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.346603 14956 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:38.346835 15015 consensus_queue.cc:1046] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:38.347604 15015 consensus_queue.cc:1225] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:38.348171 15015 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250113 08:59:38.348500 15015 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:38.234647 15146 raft_consensus.cc:738] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:38.306257 12722 rpcz_store.cc:276] 0113 08:59:32.980297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.003928 (+ 23631us) service_pool.cc:225] Handling call
0113 08:59:33.004406 (+ 478us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:33.167110 (+162704us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:33.167158 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:33.167715 (+ 557us) log.cc:844] Serialized 26 byte log entry
0113 08:59:34.550698 (+1382983us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b18003f2140. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336b2e55 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336d08c2 00007fc336285e4b 00007fc33628b913 00007fc33628bcc1 00007fc3365dd45f
0113 08:59:34.676304 (+125606us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:34.676345 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.018029 (+3341684us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54006ca7d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:38.033769 (+ 15740us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:38.046171 (+ 12402us) raft_consensus.cc:1680] finished
0113 08:59:38.046664 (+ 493us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:38.144235 (+ 97571us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1363,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":17515,"spinlock_wait_cycles":379691520,"thread_start_us":125390,"threads_started":1,"wal-append.queue_time_us":1656043,"wal-append.run_cpu_time_us":4028,"wal-append.run_wall_time_us":1327163}
I20250113 08:59:38.311722 15104 leader_election.cc:290] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.312062 14954 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:38.312439 14790 consensus_queue.cc:1046] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:38.285949 15059 consensus_queue.cc:237] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.315497 12717 raft_consensus.cc:2363] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6.
I20250113 08:59:38.294739 15065 raft_consensus.cc:738] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:38.302603 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:38.249536 15058 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.331193 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.333578 15215 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:38.335013 15247 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:38.341238 11967 rpcz_store.cc:269] 0113 08:59:35.194462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.340489 (+3146027us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.340636 (+ 147us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:38.340816 15040 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:38.326541 15259 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20250113 08:59:38.354718 11962 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:38.357949 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.360215 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:38.319644 15187 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:38.370941 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:38.351272 12686 connection.cc:232] RPC call timeout handler was delayed by -1.271278188999986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26890000000000003s for a response
I20250113 08:59:38.348899 15015 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Advancing to term 10
I20250113 08:59:38.368168 12724 raft_consensus.cc:3054] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.317957 15132 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:38.373507 12726 raft_consensus.cc:2405] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250113 08:59:38.387672 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 640) took 2478 ms. Trace:
I20250113 08:59:38.325839 15171 consensus_queue.cc:260] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:38.392068 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 543) took 3196 ms (client timeout 2787 ms). Trace:
I20250113 08:59:38.392244 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:38.392493 11967 rpcz_store.cc:269] 0113 08:59:35.195535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.387215 (+3191680us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.392009 (+ 4794us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250113 08:59:38.393236 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 161758us
W20250113 08:59:38.395227 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 154781us
W20250113 08:59:38.399075 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 155104us
I20250113 08:59:38.406504 12724 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250113 08:59:38.412940 12686 leader_election.cc:336] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.985s (SENT)
I20250113 08:59:38.389688 14790 consensus_queue.cc:1225] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250113 08:59:38.414743 12041 raft_consensus.cc:2405] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20250113 08:59:38.415256 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2313) took 2364 ms (client timeout 1961 ms). Trace:
I20250113 08:59:38.399279 11890 consensus_queue.cc:260] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.390432 15179 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:38.419277 11967 rpcz_store.cc:269] 0113 08:59:36.037769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.394393 (+2356624us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.402107 (+ 7714us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:38.421655 12686 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:38.309803 15261 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.388401 11889 rpcz_store.cc:276] 0113 08:59:35.894105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.171276 (+1277171us) service_pool.cc:225] Handling call
0113 08:59:37.283983 (+112707us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:37.284602 (+ 619us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:37.284646 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:37.285307 (+ 661us) log.cc:844] Serialized 26 byte log entry
0113 08:59:37.551024 (+265717us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:38.368796 (+817772us) log.cc:844] Serialized 14 byte log entry
0113 08:59:38.371085 (+ 2289us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.371188 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:38.372577 (+ 1389us) raft_consensus.cc:1680] finished
0113 08:59:38.372599 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:38.372772 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":562765,"prepare.queue_time_us":665338,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":1785,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":23987,"thread_start_us":507938,"threads_started":2,"wal-append.queue_time_us":466781}
W20250113 08:59:38.426141 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 211788us
I20250113 08:59:38.421027 12035 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:38.394296 14944 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:38.432072 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 528) took 3457 ms (client timeout 3194 ms). Trace:
I20250113 08:59:38.432229 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:38.432463 11967 rpcz_store.cc:269] 0113 08:59:34.974317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.427843 (+3453526us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.432013 (+ 4170us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:38.436596 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.437398 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2300) took 5621 ms. Trace:
I20250113 08:59:38.437878 11959 rpcz_store.cc:276] 0113 08:59:32.815692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.617074 (+1801382us) service_pool.cc:225] Handling call
0113 08:59:34.617523 (+ 449us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.652672 (+ 35149us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.436886 (+3784214us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b54002f0b54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:38.437188 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79321856}
W20250113 08:59:38.438412 12686 leader_election.cc:336] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.018s (SENT)
I20250113 08:59:38.438982 12686 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:38.439318 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:38.440316 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 546) took 3176 ms (client timeout 3685 ms). Trace:
W20250113 08:59:38.440472 12686 leader_election.cc:336] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.037s (SENT)
I20250113 08:59:38.443509 15256 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:38.445919 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 636) took 2635 ms. Trace:
W20250113 08:59:38.440855 11959 rpcz_store.cc:269] 0113 08:59:35.264154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.438979 (+3174825us) service_pool.cc:225] Handling call
0113 08:59:38.440229 (+ 1250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:38.441002 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:38.447568 11856 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:38.449421 15171 raft_consensus.cc:3054] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.449618 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.450007 12041 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20250113 08:59:38.450428 12723 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
W20250113 08:59:38.429219 14455 consensus_meta.cc:287] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 1.115s user 0.012s sys 0.005s
I20250113 08:59:38.447202 12722 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:38.458309 12033 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:38.459580 12719 raft_consensus.cc:2405] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20250113 08:59:38.446327 11893 rpcz_store.cc:276] 0113 08:59:35.810819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.020220 (+1209401us) service_pool.cc:225] Handling call
0113 08:59:37.086191 (+ 65971us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:37.259554 (+173363us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:37.259593 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:37.317312 (+ 57719us) log.cc:844] Serialized 26 byte log entry
0113 08:59:37.538117 (+220805us) raft_consensus.cc:1640] Marking committed up to 8
0113 08:59:38.306230 (+768113us) log.cc:844] Serialized 14 byte log entry
0113 08:59:38.322501 (+ 16271us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.322583 (+ 82us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:38.445701 (+123118us) raft_consensus.cc:1680] finished
0113 08:59:38.445721 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:38.445860 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22201,"prepare.queue_time_us":595304,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":52697,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":32494,"thread_start_us":267183,"threads_started":2,"wal-append.queue_time_us":672605}
I20250113 08:59:38.461938 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:38.468930 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:38.469031 11890 raft_consensus.cc:3054] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.461584 14944 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:38.471864 11934 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:38.478032 11856 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:38.473659 14944 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:38.479216 14944 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:38.484023 12035 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:38.484437 15263 consensus_queue.cc:1046] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250113 08:59:38.479830 11931 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:38.466534 15215 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:38.477461 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.479760 15068 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:38.488576 11967 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 529) took 3440 ms (client timeout 3433 ms). Trace:
W20250113 08:59:38.489163 11967 rpcz_store.cc:269] 0113 08:59:35.048362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.452182 (+3403820us) service_pool.cc:225] Handling call
0113 08:59:38.488473 (+ 36291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:38.488353 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.494755 14956 raft_consensus.cc:2798] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:38.506126 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:38.506805 11893 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 8.
I20250113 08:59:38.497419 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:38.498473 15173 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250113 08:59:38.501845 11960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 541) took 3307 ms (client timeout 2158 ms). Trace:
I20250113 08:59:38.501696 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:38.502490 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:38.503302 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:38.504824 15167 log.cc:926] Time spent T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 3.630s user 0.003s sys 0.003s
I20250113 08:59:38.511893 11884 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
W20250113 08:59:38.512369 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 71974.1us
I20250113 08:59:38.451134 15146 consensus_queue.cc:260] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.512533 14956 raft_consensus.cc:695] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:38.486972 14455 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:38.518422 11886 raft_consensus.cc:2405] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250113 08:59:38.513183 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:38.521234 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:38.508045 11960 rpcz_store.cc:269] 0113 08:59:35.194824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.309580 (+2114756us) service_pool.cc:225] Handling call
0113 08:59:38.501627 (+1192047us) spinlock_profiling.cc:243] Waited 6.65 ms on lock 0x7b540028f0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:38.501746 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15293184}
I20250113 08:59:38.495895 15255 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:38.525432 15104 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250113 08:59:38.525897 15104 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:38.526228 15104 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:38.527243 15019 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.527791 15019 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.529459 14790 consensus_queue.cc:1035] T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:38.507995 15173 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:38.530894 15173 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:38.532974 11959 raft_consensus.cc:3054] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.531205 15194 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.535531 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:38.536545 12040 raft_consensus.cc:2405] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250113 08:59:38.537231 12723 raft_consensus.cc:2388] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 5: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:38.533816 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.542781 11884 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 8.
I20250113 08:59:38.542793 15041 leader_election.cc:290] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:38.473997 15065 consensus_queue.cc:260] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.518131 15061 raft_consensus.cc:2798] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:59:38.545668 15021 consensus_queue.cc:1046] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250113 08:59:38.545902 15061 raft_consensus.cc:695] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:38.506019 15263 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.554885 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.516666 15058 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.555840 11960 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:38.530920 15247 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:38.569480 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 89099.6us
I20250113 08:59:38.582662 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:38.589630 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 114373us
I20250113 08:59:38.583734 12722 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:38.563254 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.599574 12723 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:38.601178 12723 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:38.607550 14956 consensus_queue.cc:237] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.608793 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:38.607787 15061 consensus_queue.cc:237] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.532182 15254 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:38.626014 15275 log.cc:926] Time spent T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20250113 08:59:38.625840 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.627317 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.628093 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 531) took 3482 ms. Trace:
I20250113 08:59:38.628417 11967 rpcz_store.cc:276] 0113 08:59:35.146263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.627008 (+3480745us) service_pool.cc:225] Handling call
0113 08:59:38.628040 (+ 1032us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:38.653304 11934 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:38.654662 12036 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:38.656158 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.595510 15224 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.660347 11934 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:38.631348 15228 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.662302 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33876 (request call id 2335) took 1008 ms. Trace:
I20250113 08:59:38.663375 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.667304 12723 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 12: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:38.687561 11963 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:38.672091 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:38.701683 12722 raft_consensus.cc:2405] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:38.709858 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.657008 15058 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:38.668040 12036 rpcz_store.cc:276] 0113 08:59:37.647198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.216041 (+568843us) service_pool.cc:225] Handling call
0113 08:59:38.655311 (+439270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:38.562587 15263 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:38.708254 15247 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:38.717095 11967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:38.719604 12725 raft_consensus.cc:1235] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
W20250113 08:59:38.722572 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 170117us
I20250113 08:59:38.723480 11960 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.730664 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:38.742797 11967 raft_consensus.cc:3054] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.667136 15251 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:38.731891 11959 raft_consensus.cc:2463] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:38.749198 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:38.743141 12039 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:38.748967 11856 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:38.752369 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.714722 15058 raft_consensus.cc:491] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:38.761684 15058 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:38.766052 15278 log.cc:926] Time spent T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250113 08:59:38.757335 15196 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:38.767222 15196 raft_consensus.cc:695] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:38.768159 11964 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2287) took 2804 ms (client timeout 1678 ms). Trace:
I20250113 08:59:38.666754 15224 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.757514 14944 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.770329 15228 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.768787 15196 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.773515 15045 raft_consensus.cc:3049] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:38.773952 15045 raft_consensus.cc:738] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:38.731567 15254 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:38.781301 12723 raft_consensus.cc:3054] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:38.756352 12686 leader_election.cc:336] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.158s (SENT)
W20250113 08:59:38.768630 11964 rpcz_store.cc:269] 0113 08:59:35.964124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.767934 (+2803810us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.768078 (+ 144us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:38.760149 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:38.760321 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2279) took 3585 ms (client timeout 2359 ms). Trace:
I20250113 08:59:38.760475 12036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5"
dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
from {username='slave'} at 127.0.0.1:37184
I20250113 08:59:38.760794 15247 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:38.766178 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2262) took 5799 ms. Trace:
W20250113 08:59:38.804572 11968 rpcz_store.cc:269] 0113 08:59:35.174914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.760060 (+3585146us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:38.760227 (+ 167us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:38.759182 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:38.723026 14963 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:38.816901 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 885) took 5213 ms. Trace:
I20250113 08:59:38.810243 12036 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:59:38.822999 12036 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.825603 12686 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:38.821375 12725 rpcz_store.cc:276] 0113 08:59:33.603372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.083371 (+1479999us) service_pool.cc:225] Handling call
0113 08:59:38.718767 (+3635396us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b5800238e98. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc333693142 00007fc33c363cd9 00007fc33b4f2750 00007fc33b5b87fb 00007fc33b5b8881 00007fc33b57c2a0 00007fc33b5677e9 00007fc33b567292 00007fc334c33694
0113 08:59:38.719058 (+ 291us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:38.754252 (+ 35194us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.816809 (+ 62557us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11178880}
I20250113 08:59:38.820042 11966 rpcz_store.cc:276] 0113 08:59:32.967250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.728727 (+1761477us) service_pool.cc:225] Handling call
0113 08:59:37.129409 (+2400682us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:37.228934 (+ 99525us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.765956 (+1537022us) spinlock_profiling.cc:243] Waited 2.34 s on lock 0x7b54001ae5d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:38.766113 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5381495424}
I20250113 08:59:38.734431 15261 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:38.872409 12720 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 674) took 4555 ms (client timeout 2199 ms). Trace:
I20250113 08:59:38.881757 12718 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:38.877164 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.876855 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.853053 14944 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:38.893880 11960 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:38.895138 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.895983 11960 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 8.
I20250113 08:59:38.823038 14963 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:38.886647 11964 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:38.885660 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:38.876689 12720 rpcz_store.cc:269] 0113 08:59:34.317313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.769883 (+452570us) service_pool.cc:225] Handling call
0113 08:59:38.798929 (+4029046us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400501254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:38.872143 (+ 73214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2674559744}
I20250113 08:59:38.905637 11966 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:38.913947 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:38.925690 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.855059 15015 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:38.942989 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 186697us
I20250113 08:59:38.943112 15045 consensus_queue.cc:260] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:38.904457 14963 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:38.950407 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.954164 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.978216 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:38.981693 12007 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:38.981376 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43872 (request call id 2316) took 1522 ms. Trace:
I20250113 08:59:38.941291 15261 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 08:59:38.985620 11966 rpcz_store.cc:276] 0113 08:59:37.459003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.926848 (+1467845us) service_pool.cc:225] Handling call
0113 08:59:38.981215 (+ 54367us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:38.977429 12686 leader_election.cc:336] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.223s (SENT)
I20250113 08:59:38.986716 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:38.998724 12035 raft_consensus.cc:2388] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:38.974879 14845 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:38.985489 14992 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
W20250113 08:59:39.026768 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 75183.9us
I20250113 08:59:39.027498 14992 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.029383 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.030325 12041 raft_consensus.cc:2405] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250113 08:59:39.031260 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.032455 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.034317 12033 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:39.046697 15040 consensus_meta.cc:287] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Time spent flushing consensus metadata: real 0.664s user 0.011s sys 0.009s
I20250113 08:59:38.997972 15224 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.063746 12686 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:39.029045 11858 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:39.070529 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 115430us
I20250113 08:59:39.071281 12007 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:39.071974 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2267) took 5654 ms. Trace:
I20250113 08:59:39.072511 11968 rpcz_store.cc:276] 0113 08:59:33.356861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.947204 (+5590343us) service_pool.cc:225] Handling call
0113 08:59:38.990753 (+ 43549us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.010553 (+ 19800us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:39.010621 (+ 68us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:39.010685 (+ 64us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:39.010749 (+ 64us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:39.010847 (+ 98us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:39.011025 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2117}
I20250113 08:59:38.960144 14944 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:39.077015 12036 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.092443 12722 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:39.091905 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 62925.4us
I20250113 08:59:39.094846 12723 raft_consensus.cc:2463] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:39.103232 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.122843 12007 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:39.164196 12041 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 11.
W20250113 08:59:39.148636 11890 consensus_meta.cc:287] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.677s user 0.012s sys 0.003s
I20250113 08:59:39.166918 11967 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:39.106483 12035 raft_consensus.cc:1270] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250113 08:59:39.184793 11968 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:39.185700 11966 raft_consensus.cc:1235] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:39.182934 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 91350.8us
I20250113 08:59:39.095372 15003 raft_consensus.cc:2798] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:39.193163 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2303) took 5671 ms. Trace:
I20250113 08:59:39.193773 11968 rpcz_store.cc:276] 0113 08:59:33.521636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.074125 (+5552489us) service_pool.cc:225] Handling call
0113 08:59:39.168598 (+ 94473us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.192832 (+ 24234us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:39.193080 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:39.121704 14953 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:39.122505 14992 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.194388 12007 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.114879 15263 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:39.018154 15173 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.102634 15179 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250113 08:59:39.021487 15104 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:39.108721 15091 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:39.084833 14845 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:39.222234 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:39.217375 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 123163us
I20250113 08:59:39.195351 14953 raft_consensus.cc:491] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:39.089424 15053 consensus_queue.cc:1046] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.105s
I20250113 08:59:39.207926 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:39.213773 15289 log.cc:926] Time spent T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
W20250113 08:59:39.224015 11887 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 625) took 6652 ms (client timeout 2787 ms). Trace:
I20250113 08:59:39.258922 11893 raft_consensus.cc:2388] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:39.224051 11968 raft_consensus.cc:1235] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:39.191365 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.125083 14944 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:39.219426 15161 raft_consensus.cc:3049] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:39.231925 15179 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:39.222771 15058 raft_consensus.cc:513] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.296145 15179 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:39.218751 15001 raft_consensus.cc:3049] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:39.298079 15001 raft_consensus.cc:738] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:39.299100 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2272) took 4658 ms. Trace:
W20250113 08:59:39.220208 15253 log.cc:926] Time spent T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 1.196s user 0.000s sys 0.008s
I20250113 08:59:39.217649 15035 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:39.300913 15035 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:39.301646 15035 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:39.264066 11887 rpcz_store.cc:269] 0113 08:59:32.571169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:33.441329 (+870160us) service_pool.cc:225] Handling call
0113 08:59:39.162858 (+5721529us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540012f054. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:39.223831 (+ 60973us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2735170432}
I20250113 08:59:39.303062 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:39.303871 14953 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:39.217077 15286 log.cc:926] Time spent T 1527f41e7e954abc8622a5f5da292602 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.063s user 0.000s sys 0.003s
I20250113 08:59:39.291510 14956 consensus_queue.cc:1035] T a43de3c8f06b48358bcee58f9567bb3b P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20250113 08:59:39.242082 11858 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:39.217787 15173 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.304744 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.299726 11968 rpcz_store.cc:276] 0113 08:59:34.641054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.206731 (+4565677us) service_pool.cc:225] Handling call
0113 08:59:39.223772 (+ 17041us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.298723 (+ 74951us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:39.299008 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:39.312033 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.295549 15161 raft_consensus.cc:738] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:39.237003 11967 raft_consensus.cc:1212] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Deduplicated request from leader. Original: 2.5->[3.6-5.7] Dedup: 3.6->[5.7-5.7]
I20250113 08:59:39.353885 11967 raft_consensus.cc:1235] T 07a14a3957e44925ae0925b645d220eb P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
W20250113 08:59:39.355981 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 113999us
I20250113 08:59:39.360236 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.363389 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2271) took 4714 ms. Trace:
I20250113 08:59:39.363999 11967 rpcz_store.cc:276] 0113 08:59:34.640812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.190380 (+4549568us) service_pool.cc:225] Handling call
0113 08:59:39.236800 (+ 46420us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:39.355281 (+118481us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:39.355499 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:39.225656 15003 raft_consensus.cc:695] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:39.369804 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.372735 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.370784 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1896) took 2195 ms. Trace:
I20250113 08:59:39.366997 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.373845 11890 rpcz_store.cc:276] 0113 08:59:37.175276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.231666 (+1056390us) service_pool.cc:225] Handling call
0113 08:59:38.255352 (+ 23686us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.362856 (+1107504us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:39.363065 (+ 209us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:39.363150 (+ 85us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:39.370412 (+ 7262us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:39.370529 (+ 117us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:39.370709 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":38640,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":169018,"threads_started":1}
I20250113 08:59:39.376365 11968 raft_consensus.cc:1270] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250113 08:59:39.311007 15001 consensus_queue.cc:260] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:39.294052 14845 raft_consensus.cc:513] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:39.380095 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 68557.8us
I20250113 08:59:39.380574 15302 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:39.381740 14944 leader_election.cc:290] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:39.383873 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:39.367051 15053 consensus_queue.cc:1225] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250113 08:59:39.384727 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.393651 12035 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:39.385521 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2273) took 4744 ms. Trace:
I20250113 08:59:39.394909 11968 rpcz_store.cc:276] 0113 08:59:34.641293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.318443 (+4677150us) service_pool.cc:225] Handling call
0113 08:59:39.325591 (+ 7148us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.379026 (+ 53435us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:39.385428 (+ 6402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:39.353660 15058 leader_election.cc:290] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.368885 14956 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250113 08:59:39.402105 15104 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:39.397341 15199 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:39.401520 14845 leader_election.cc:290] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.423348 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.423830 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:39.424408 11887 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:39.426064 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.426944 12035 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 11 FOLLOWER]: Advancing to term 12
I20250113 08:59:39.436041 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.380606 15003 consensus_queue.cc:237] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.443377 15196 consensus_queue.cc:1046] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250113 08:59:39.377928 15015 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 10 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.414742 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:39.402487 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 219926us
W20250113 08:59:39.381067 15287 log.cc:926] Time spent T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
W20250113 08:59:39.444497 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 47556.5us
I20250113 08:59:39.446619 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:39.453348 11968 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:39.448016 11931 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.383203 14963 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.456929 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.459131 11893 raft_consensus.cc:2388] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:39.463678 15196 consensus_queue.cc:1225] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250113 08:59:39.464327 15196 raft_consensus.cc:3049] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:39.464660 15196 raft_consensus.cc:738] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:39.460369 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.465971 11931 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.473795 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.458492 15001 raft_consensus.cc:3054] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:39.453087 15295 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:39.479935 11887 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:39.485450 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.437181 15302 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.487879 11887 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:39.489511 15302 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.489658 14963 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.489199 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:39.490952 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.492858 11887 raft_consensus.cc:2463] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:39.486280 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.494866 11893 raft_consensus.cc:2388] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 9: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:39.457266 15161 consensus_queue.cc:260] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.724753 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.549348 12035 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 12.
I20250113 08:59:39.752074 12039 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Advancing to term 10
I20250113 08:59:39.743044 12006 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:39.758318 11968 raft_consensus.cc:2463] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:39.549204 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.788111 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:39.566468 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 51514.5us
I20250113 08:59:39.752204 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.497944 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.811599 12035 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 9
I20250113 08:59:39.789916 15173 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:39.566212 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 377904us
I20250113 08:59:39.811734 12041 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
W20250113 08:59:39.791389 12685 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.677s (SENT) [suppressed 35 similar messages]
I20250113 08:59:39.815661 12007 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:39.761036 15058 raft_consensus.cc:2743] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:39.824182 15173 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.849134 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.494110 14845 raft_consensus.cc:2743] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:39.860826 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:39.880043 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.889458 11968 raft_consensus.cc:2388] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:39.885637 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:39.881592 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 65985.2us
I20250113 08:59:39.884449 11887 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:39.890903 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.891927 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1904) took 1166 ms. Trace:
I20250113 08:59:39.891227 11859 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.892346 11887 rpcz_store.cc:276] 0113 08:59:38.726043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.792145 (+1066102us) service_pool.cc:225] Handling call
0113 08:59:39.891864 (+ 99719us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:39.894081 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 151217us
I20250113 08:59:39.821591 15003 raft_consensus.cc:2743] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:39.723749 15299 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.340s
I20250113 08:59:39.897846 12039 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 10.
I20250113 08:59:39.899593 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:39.497828 14963 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.394s
I20250113 08:59:39.881599 15279 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:39.906060 11856 leader_election.cc:304] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:39.884418 15305 raft_consensus.cc:2798] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:39.912410 12035 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9.
I20250113 08:59:39.913528 12041 raft_consensus.cc:2388] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:39.913686 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.914686 11856 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:39.898391 15104 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:39.911197 12006 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:39.550383 15196 consensus_queue.cc:260] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:39.905735 14944 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:39.919206 11931 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.917691 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.905949 14845 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:39.945364 12039 raft_consensus.cc:2388] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:39.945370 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 128378us
I20250113 08:59:39.947664 11859 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.949302 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.917856 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:39.952778 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 161426us
I20250113 08:59:39.954123 11887 raft_consensus.cc:2405] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250113 08:59:39.956337 15310 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:39.944829 15130 raft_consensus.cc:2798] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 10 FOLLOWER]: Leader election won for term 10
W20250113 08:59:39.824702 15053 consensus_queue.cc:1159] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" responder_term: 8 status { last_received { term: 6 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [7.9-7.9]" } } last_received_current_leader { term: 0 index: 0 } }
I20250113 08:59:39.957784 14956 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader election won for term 9
I20250113 08:59:39.957886 15196 raft_consensus.cc:3054] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:39.958847 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 64233.6us
I20250113 08:59:39.959687 12006 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:39.960800 15121 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:39.953760 14845 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.963734 15102 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:39.964363 15102 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:39.964517 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.961402 15121 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:39.963809 14845 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.966961 11859 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.967756 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:39.918433 15104 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:39.907207 15173 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:39.979678 15104 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:39.950078 15003 raft_consensus.cc:491] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:39.992789 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 78182.9us
I20250113 08:59:39.994967 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:39.995682 11887 raft_consensus.cc:2363] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7.
I20250113 08:59:39.966836 14992 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250113 08:59:39.919788 15199 consensus_queue.cc:1035] T 49d70ffc378340e785c4f8dde909f9ba P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20250113 08:59:39.999840 11859 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:39.919147 15279 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:39.998580 15001 consensus_meta.cc:287] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.520s user 0.010s sys 0.000s
I20250113 08:59:39.993041 14944 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:40.002885 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa96abb4c9940e4aa27327deeb68af0" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:39.932199 15304 consensus_queue.cc:1046] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20250113 08:59:40.000664 14956 raft_consensus.cc:695] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [term 9 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:39.998792 14992 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:40.010308 11968 raft_consensus.cc:2405] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250113 08:59:40.010918 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1773) took 2908 ms. Trace:
I20250113 08:59:40.012112 12042 rpcz_store.cc:276] 0113 08:59:37.103140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.117778 (+ 14638us) service_pool.cc:225] Handling call
0113 08:59:40.002723 (+2884945us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b58002d1898. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333693291 00007fc33369360e 0000000000353db9 00007fc33b4f2a50 00007fc33b5b8814 00007fc33b5b8881 00007fc33b5681a0 00007fc33b567a84 00007fc334c35c14 00007fc334c35ba2
0113 08:59:40.010866 (+ 8143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6270336}
W20250113 08:59:40.012665 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 100604us
I20250113 08:59:40.013043 15309 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:40.023574 15309 raft_consensus.cc:513] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.014931 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2323) took 4593 ms. Trace:
I20250113 08:59:39.996093 15003 raft_consensus.cc:513] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.014880 15121 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.015002 12042 raft_consensus.cc:1270] T 67a8b82ef293439e815f49903f8e5dbb P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250113 08:59:40.016026 15307 log.cc:926] Time spent T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250113 08:59:40.006954 15130 raft_consensus.cc:695] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 10 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:40.026063 15309 leader_election.cc:290] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.064122 15279 leader_election.cc:290] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.052809 12006 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.015831 14956 consensus_queue.cc:237] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.019189 11931 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.009008 14992 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Advancing to term 9
W20250113 08:59:40.023025 14953 consensus_meta.cc:287] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.718s user 0.013s sys 0.006s
I20250113 08:59:40.013854 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.025882 12037 rpcz_store.cc:276] 0113 08:59:35.421572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.592231 (+170659us) service_pool.cc:225] Handling call
0113 08:59:35.720220 (+127989us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:35.723236 (+ 3016us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.014395 (+4291159us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b54002b9150. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.014704 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65082752}
I20250113 08:59:40.055989 14944 raft_consensus.cc:2743] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 08:59:40.051162 15304 raft_consensus.cc:3049] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:40.093392 15304 raft_consensus.cc:738] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:40.094528 15304 consensus_queue.cc:260] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.093648 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.097803 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.081638 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 624) took 7522 ms. Trace:
I20250113 08:59:40.081128 15130 consensus_queue.cc:237] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.082873 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.084369 15151 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:40.072001 14953 raft_consensus.cc:513] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.085649 15279 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250113 08:59:40.089568 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 70446.6us
I20250113 08:59:40.051115 14963 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:40.050899 15311 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:40.102025 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 105404us
I20250113 08:59:40.116755 15282 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250113 08:59:40.106292 15304 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:40.099136 11888 rpcz_store.cc:276] 0113 08:59:32.558755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.883321 (+2324566us) service_pool.cc:225] Handling call
0113 08:59:34.896360 (+ 13039us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:34.968714 (+ 72354us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:34.968749 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:34.968797 (+ 48us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:34.968845 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.081210 (+5112365us) spinlock_profiling.cc:243] Waited 72.1 ms on lock 0x7b540026c854. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:40.081297 (+ 87us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.081479 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165857536}
I20250113 08:59:40.108237 12042 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:40.121809 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.123376 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.124159 11888 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:40.110392 11893 raft_consensus.cc:2388] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:40.126801 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1910) took 1223 ms. Trace:
I20250113 08:59:40.127342 15104 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.115939 14963 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.125187 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.108992 14992 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.117529 11859 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.127156 11893 rpcz_store.cc:276] 0113 08:59:38.903679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.501504 (+597825us) service_pool.cc:225] Handling call
0113 08:59:40.126580 (+625076us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b540026cd54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.126719 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":753415552}
W20250113 08:59:40.176307 11959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2289) took 2986 ms (client timeout 2230 ms). Trace:
I20250113 08:59:40.122006 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.105362 15305 raft_consensus.cc:695] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:40.117566 15035 consensus_meta.cc:287] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.815s user 0.012s sys 0.001s
W20250113 08:59:40.181114 11884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 2004) took 1673 ms (client timeout 1624 ms). Trace:
W20250113 08:59:40.181635 11884 rpcz_store.cc:269] 0113 08:59:38.500504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.512832 (+ 12328us) service_pool.cc:225] Handling call
0113 08:59:40.130119 (+1617287us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b540038ed54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.174144 (+ 44025us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48540928}
I20250113 08:59:40.173730 12724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 696) took 2609 ms. Trace:
I20250113 08:59:40.182399 12724 rpcz_store.cc:276] 0113 08:59:37.564535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.249651 (+685116us) service_pool.cc:225] Handling call
0113 08:59:38.281047 (+ 31396us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:38.407131 (+126084us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.129742 (+1722611us) spinlock_profiling.cc:243] Waited 79.8 ms on lock 0x7b54004105d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:40.173649 (+ 43907us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183610752}
I20250113 08:59:40.174381 12037 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250113 08:59:40.183741 11888 raft_consensus.cc:2388] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 11: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:40.129141 15044 raft_consensus.cc:3054] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:40.167110 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 62298.5us
I20250113 08:59:40.183593 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.185935 11893 raft_consensus.cc:2405] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:40.118244 14953 leader_election.cc:290] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:40.176709 11959 rpcz_store.cc:269] 0113 08:59:37.189636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.455699 (+1266063us) service_pool.cc:225] Handling call
0113 08:59:40.145185 (+1689486us) spinlock_profiling.cc:243] Waited 63.4 ms on lock 0x7b54001a13d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.176235 (+ 31050us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145771776}
W20250113 08:59:40.178737 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 147433us
I20250113 08:59:40.189085 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 631) took 1098 ms. Trace:
I20250113 08:59:40.188881 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:40.179059 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 900) took 2841 ms (client timeout 1996 ms). Trace:
I20250113 08:59:38.622088 10702 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250113 08:59:40.180223 12725 raft_consensus.cc:1235] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:40.189548 12035 rpcz_store.cc:276] 0113 08:59:39.094916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.977170 (+882254us) service_pool.cc:225] Handling call
0113 08:59:39.977610 (+ 440us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.978111 (+ 501us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:40.008208 (+ 30097us) log.cc:844] Serialized 14 byte log entry
0113 08:59:40.188708 (+180500us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:40.188755 (+ 47us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:40.188821 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.188886 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.189016 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":47030,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":764,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":201187,"threads_started":2,"wal-append.queue_time_us":178450}
I20250113 08:59:40.190390 11893 raft_consensus.cc:2405] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250113 08:59:40.190622 12717 rpcz_store.cc:269] 0113 08:59:37.337963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.486886 (+148923us) service_pool.cc:225] Handling call
0113 08:59:40.138645 (+2651759us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b54006c9654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.178982 (+ 40337us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1762306304}
I20250113 08:59:40.201522 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.191556 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52600 (request call id 1911) took 1214 ms. Trace:
I20250113 08:59:40.203716 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.192127 12035 raft_consensus.cc:1270] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:40.184058 15035 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.194172 14992 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:40.194731 12042 raft_consensus.cc:1270] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250113 08:59:40.180663 15299 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:40.186668 15118 raft_consensus.cc:491] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:40.196296 15161 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:40.196539 15176 log.cc:926] Time spent T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 5.229s user 0.012s sys 0.011s
I20250113 08:59:40.196077 15305 consensus_queue.cc:237] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.199216 11856 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:40.204138 11893 rpcz_store.cc:276] 0113 08:59:38.977581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.188577 (+1210996us) service_pool.cc:225] Handling call
0113 08:59:40.191480 (+ 2903us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.179107 11887 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:40.197072 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.217183 12035 raft_consensus.cc:1270] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250113 08:59:40.217001 12037 raft_consensus.cc:3049] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:40.202565 11884 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 8.
W20250113 08:59:40.184130 15179 consensus_meta.cc:287] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.887s user 0.024s sys 0.000s
I20250113 08:59:40.211141 15104 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:40.211382 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 93938.1us
I20250113 08:59:40.211591 15247 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.208626 14783 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.219138 11859 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.220642 11859 leader_election.cc:304] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.223176 11859 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.223925 15104 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:40.224696 15104 raft_consensus.cc:2743] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 08:59:40.225339 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.226105 15019 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.214229 15118 raft_consensus.cc:513] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.226969 11888 raft_consensus.cc:1235] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:40.227553 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2008) took 1660 ms. Trace:
I20250113 08:59:40.228047 14963 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.228199 15118 leader_election.cc:290] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.211637 14664 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.020s
I20250113 08:59:40.227857 11888 rpcz_store.cc:276] 0113 08:59:38.566810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.226408 (+1659598us) service_pool.cc:225] Handling call
0113 08:59:40.226773 (+ 365us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.227330 (+ 557us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.227489 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.217990 12037 raft_consensus.cc:738] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:40.231019 11892 raft_consensus.cc:2405] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:40.224931 11859 leader_election.cc:304] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.231875 12035 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:40.233220 15322 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:40.233703 15322 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:40.234169 15322 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:40.233775 11884 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:40.241796 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1901) took 1568 ms. Trace:
I20250113 08:59:40.242328 11884 rpcz_store.cc:276] 0113 08:59:38.672960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.233210 (+1560250us) service_pool.cc:225] Handling call
0113 08:59:40.233597 (+ 387us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.241436 (+ 7839us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.241711 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.225389 15104 raft_consensus.cc:2743] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:40.244139 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 614) took 6800 ms. Trace:
I20250113 08:59:40.244979 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.246948 11893 raft_consensus.cc:2405] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250113 08:59:40.247411 12042 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250113 08:59:40.248793 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.250119 11959 raft_consensus.cc:2388] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:40.254552 14963 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:40.255463 14963 consensus_queue.cc:1225] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 5e8883f5f83a47f4b9303dd1da4f59c0 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:40.255935 14990 consensus_queue.cc:1035] T 8210eb7687124bcfbc64e314e532799e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20250113 08:59:40.256103 15058 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:40.255249 14790 consensus_queue.cc:1035] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250113 08:59:40.253696 15003 leader_election.cc:290] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.264158 11891 raft_consensus.cc:1235] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:40.265015 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1903) took 1542 ms. Trace:
I20250113 08:59:40.268159 11891 rpcz_store.cc:276] 0113 08:59:38.722356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.263534 (+1541178us) service_pool.cc:225] Handling call
0113 08:59:40.263935 (+ 401us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.264723 (+ 788us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.264949 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.274241 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.275111 11959 raft_consensus.cc:2463] T 80cc6c97986444348b1837c27c41338a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:40.284058 11891 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:40.200013 15102 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.266710 12037 consensus_queue.cc:260] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.288450 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1905) took 1546 ms. Trace:
I20250113 08:59:40.290021 11891 rpcz_store.cc:276] 0113 08:59:38.742099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.269811 (+1527712us) service_pool.cc:225] Handling call
0113 08:59:40.283828 (+ 14017us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.288094 (+ 4266us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.288384 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.244616 12038 rpcz_store.cc:276] 0113 08:59:33.443214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.627630 (+1184416us) service_pool.cc:225] Handling call
0113 08:59:34.628028 (+ 398us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:34.628472 (+ 444us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:34.628505 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:34.629204 (+ 699us) log.cc:844] Serialized 52 byte log entry
0113 08:59:34.698901 (+ 69697us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:34.698940 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:34.699016 (+ 76us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:40.130458 (+5431442us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:40.229558 (+ 99100us) raft_consensus.cc:1680] finished
0113 08:59:40.229579 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.243973 (+ 14394us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11325,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":92,"spinlock_wait_cycles":7839488,"thread_start_us":53236,"threads_started":1,"wal-append.queue_time_us":260834}
I20250113 08:59:40.293731 12037 raft_consensus.cc:3054] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:40.293836 11889 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57686 (request call id 646) took 1866 ms (client timeout 1677 ms). Trace:
I20250113 08:59:40.294068 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:40.294277 11889 rpcz_store.cc:269] 0113 08:59:38.427801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.293627 (+1865826us) service_pool.cc:212] Skipping call since client already timed out
0113 08:59:40.293773 (+ 146us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250113 08:59:40.294888 12038 raft_consensus.cc:2388] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:40.295184 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.295696 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.296381 15035 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:40.297138 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1902) took 1586 ms. Trace:
I20250113 08:59:40.297542 11884 rpcz_store.cc:276] 0113 08:59:38.713710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.243294 (+1529584us) service_pool.cc:225] Handling call
0113 08:59:40.276177 (+ 32883us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.296755 (+ 20578us) raft_consensus.cc:1529] Early marking committed up to index 3
0113 08:59:40.296785 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:40.296830 (+ 45us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:40.296868 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.296936 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.297054 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1167}
W20250113 08:59:40.298383 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 59581.2us
I20250113 08:59:40.299130 11859 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:40.295841 11889 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:40.296396 12038 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:40.308604 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1779) took 1549 ms. Trace:
I20250113 08:59:40.309046 12033 rpcz_store.cc:276] 0113 08:59:38.759668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.759827 (+ 159us) service_pool.cc:225] Handling call
0113 08:59:39.034136 (+274309us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.034755 (+ 619us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.308396 (+1273641us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54002b0550. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.308544 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":630074112}
I20250113 08:59:40.310299 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:40.310878 12033 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:40.267834 15247 raft_consensus.cc:2743] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:40.317407 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.318917 12036 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.319895 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.321853 11893 raft_consensus.cc:1270] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250113 08:59:40.321328 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.231045 15179 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.293316 15102 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.323073 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.324349 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:40.322225 11893 pending_rounds.cc:77] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 2
I20250113 08:59:40.325198 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 628) took 1252 ms. Trace:
I20250113 08:59:40.325690 12039 rpcz_store.cc:276] 0113 08:59:39.072303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.965666 (+893363us) service_pool.cc:225] Handling call
0113 08:59:39.966018 (+ 352us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.966313 (+ 295us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:40.068424 (+102111us) log.cc:844] Serialized 14 byte log entry
0113 08:59:40.324792 (+256368us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:40.324855 (+ 63us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:40.324925 (+ 70us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.325011 (+ 86us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.325141 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7201,"prepare.queue_time_us":58597,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":252351,"threads_started":2,"wal-append.queue_time_us":215248}
I20250113 08:59:40.325337 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1786) took 1255 ms. Trace:
I20250113 08:59:40.327448 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.325233 11893 pending_rounds.cc:99] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250113 08:59:40.323861 11959 raft_consensus.cc:3054] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:40.330122 12033 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:40.323252 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:37184 (request call id 625) took 1560 ms. Trace:
I20250113 08:59:40.331718 12036 rpcz_store.cc:276] 0113 08:59:38.759298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.760326 (+ 1028us) service_pool.cc:225] Handling call
0113 08:59:40.315572 (+1555246us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b540020d5d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336600da7 00007fc33b569ed0 00007fc334c41794 00007fc334c41722 00007fc334c41651 00007fc334c415cc 00007fc334c40882 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.319369 (+ 3797us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":230,"spinlock_wait_cycles":12458880}
I20250113 08:59:40.339102 12036 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250113 08:59:40.340796 12036 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:40.279887 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 262515us
I20250113 08:59:40.330054 11893 raft_consensus.cc:2883] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:40.328073 12041 rpcz_store.cc:276] 0113 08:59:39.070171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.967634 (+897463us) service_pool.cc:225] Handling call
0113 08:59:39.989682 (+ 22048us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.997452 (+ 7770us) raft_consensus.cc:1529] Early marking committed up to index 8
0113 08:59:40.065579 (+ 68127us) log.cc:844] Serialized 14 byte log entry
0113 08:59:40.275465 (+209886us) log.cc:844] Serialized 14 byte log entry
0113 08:59:40.306186 (+ 30721us) log.cc:844] Serialized 14 byte log entry
0113 08:59:40.324426 (+ 18240us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:40.324479 (+ 53us) raft_consensus.cc:1640] Marking committed up to 8
0113 08:59:40.324621 (+ 142us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.325120 (+ 499us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.325244 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25193,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":71,"raft.queue_time_us":24359,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":9604,"spinlock_wait_cycles":47868160,"thread_start_us":204936,"threads_started":2,"wal-append.queue_time_us":210361}
W20250113 08:59:40.353742 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 54696.3us
I20250113 08:59:40.332083 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.358525 11893 pending_rounds.cc:99] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250113 08:59:40.354593 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.359333 11893 raft_consensus.cc:2883] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:40.366042 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.366745 11959 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9.
I20250113 08:59:40.366945 11889 raft_consensus.cc:3054] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 11 FOLLOWER]: Advancing to term 12
I20250113 08:59:40.368005 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.363586 11856 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:40.370046 12041 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 6. Current term is 7. Ops: [3.3-6.4]
I20250113 08:59:40.371177 12033 raft_consensus.cc:2405] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250113 08:59:40.372470 11884 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 4.
I20250113 08:59:40.372701 12006 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.348271 15313 raft_consensus.cc:2743] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:40.374570 12042 raft_consensus.cc:1235] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 8. Current term is 9. Ops: []
I20250113 08:59:40.376358 11931 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:40.377681 11859 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.377918 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1906) took 1597 ms. Trace:
I20250113 08:59:40.378225 11893 rpcz_store.cc:276] 0113 08:59:38.775387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.277377 (+1501990us) service_pool.cc:225] Handling call
0113 08:59:40.298902 (+ 21525us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.372162 (+ 73260us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.372351 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.381098 11858 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:40.360339 15247 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:40.386103 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d1890dc82d4bd889d0300c9fed050d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.387557 12038 raft_consensus.cc:1270] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:40.383517 15021 consensus_queue.cc:1046] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250113 08:59:40.389175 11891 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:40.389535 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.390156 11959 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:40.390976 11858 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:40.378717 15102 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:40.398480 12037 raft_consensus.cc:1270] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250113 08:59:40.382740 15130 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
W20250113 08:59:40.399580 11859 connection.cc:232] RPC call timeout handler was delayed by -1.2650198490000548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
W20250113 08:59:40.437657 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 94377.8us
I20250113 08:59:40.439141 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2268) took 7039 ms. Trace:
I20250113 08:59:40.440404 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.384778 14664 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:40.439682 11966 rpcz_store.cc:276] 0113 08:59:33.358051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.026286 (+5668235us) service_pool.cc:225] Handling call
0113 08:59:39.169233 (+142947us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:39.190018 (+ 20785us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.397618 (+1207600us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54001adbd0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.397715 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":327628160}
W20250113 08:59:40.441833 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 83181.1us
I20250113 08:59:40.382830 15035 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:40.385288 15247 raft_consensus.cc:491] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:40.443446 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.400277 15021 raft_consensus.cc:3049] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:40.406935 12042 pending_rounds.cc:77] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 4
I20250113 08:59:40.482429 12042 pending_rounds.cc:99] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250113 08:59:40.482864 12042 raft_consensus.cc:2883] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:40.443816 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.483632 11891 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.444696 15102 raft_consensus.cc:695] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:40.447101 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.393435 15031 consensus_queue.cc:1046] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20250113 08:59:40.449687 15179 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:40.449764 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1900) took 1779 ms. Trace:
I20250113 08:59:40.487823 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.393219 15015 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:40.442530 12006 leader_election.cc:304] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.495442 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.397392 14956 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
W20250113 08:59:40.502765 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 51163.2us
I20250113 08:59:40.501883 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.389613 15058 consensus_queue.cc:1046] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20250113 08:59:40.393576 15019 consensus_queue.cc:1035] T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
I20250113 08:59:40.454811 14664 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:40.506911 11932 leader_election.cc:304] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:40.507043 11966 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:40.500238 11888 rpcz_store.cc:276] 0113 08:59:38.670007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.235048 (+1565041us) service_pool.cc:225] Handling call
0113 08:59:40.235478 (+ 430us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:40.248929 (+ 13451us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:40.248975 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:40.297410 (+ 48435us) log.cc:844] Serialized 26 byte log entry
0113 08:59:40.387385 (+ 89975us) raft_consensus.cc:1640] Marking committed up to 5
0113 08:59:40.387423 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.387482 (+ 59us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:40.449513 (+ 62031us) raft_consensus.cc:1680] finished
0113 08:59:40.449534 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.449682 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":373,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9131,"thread_start_us":89541,"threads_started":1,"wal-append.queue_time_us":102565}
I20250113 08:59:40.510104 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.488472 15021 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:40.513742 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.514662 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.492684 15247 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:40.515061 11959 raft_consensus.cc:2371] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7. Re-sending same reply.
I20250113 08:59:40.515230 11893 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.438952 14963 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20250113 08:59:40.518395 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.519280 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.510804 14664 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.520293 15031 raft_consensus.cc:3049] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Stepping down as leader of term 8
I20250113 08:59:40.522037 12039 raft_consensus.cc:2405] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250113 08:59:40.524467 11932 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.529999 11932 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.530838 11932 leader_election.cc:304] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:40.531666 11932 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.549947 11967 raft_consensus.cc:1270] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250113 08:59:40.555804 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.556618 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.558203 11888 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:40.519129 15323 raft_consensus.cc:2743] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250113 08:59:40.569473 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 126977us
I20250113 08:59:40.533847 15019 raft_consensus.cc:2798] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:40.571372 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 551) took 1189 ms. Trace:
I20250113 08:59:40.572614 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.541204 15279 raft_consensus.cc:2758] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250113 08:59:40.524932 15031 raft_consensus.cc:738] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:40.541082 15282 raft_consensus.cc:2798] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:40.576764 15282 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:40.577193 15282 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.555989 15273 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:40.573869 11959 rpcz_store.cc:276] 0113 08:59:39.381452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.525931 (+1144479us) service_pool.cc:225] Handling call
0113 08:59:40.570995 (+ 45064us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.547973 14790 raft_consensus.cc:2743] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250113 08:59:40.565652 15058 raft_consensus.cc:3054] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.593367 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47a39083de44711b040bcf422953893" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.567281 15223 consensus_queue.cc:1046] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.021s
I20250113 08:59:40.584293 15204 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250113 08:59:40.597324 11859 leader_election.cc:336] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.996s (SENT)
I20250113 08:59:40.605372 11858 leader_election.cc:304] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:40.606226 11932 leader_election.cc:304] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.608278 12006 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.589704 15019 raft_consensus.cc:695] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:40.609856 11859 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.610510 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.580663 15044 consensus_queue.cc:1035] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.018s
I20250113 08:59:40.614218 14723 raft_consensus.cc:2743] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250113 08:59:40.614559 15333 log.cc:926] Time spent T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20250113 08:59:40.619421 15323 raft_consensus.cc:2743] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:40.619856 12006 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.605036 15204 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:40.620998 15060 consensus_queue.cc:1046] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250113 08:59:40.599628 15309 raft_consensus.cc:491] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:59:40.620703 15334 log.cc:926] Time spent T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20250113 08:59:40.584370 15273 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.574054 15021 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.635648 15204 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:40.660925 15019 consensus_queue.cc:237] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.581156 15102 consensus_queue.cc:237] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.642729 15168 consensus_queue.cc:260] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.643669 12006 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.669468 15168 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:40.644246 14879 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.644739 15309 raft_consensus.cc:513] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.622272 14963 raft_consensus.cc:2743] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:40.654691 15338 log.cc:926] Time spent T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20250113 08:59:40.684667 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 87367.7us
I20250113 08:59:40.631832 15059 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:40.674649 11889 raft_consensus.cc:2463] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 12.
I20250113 08:59:40.681335 15060 consensus_queue.cc:1225] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250113 08:59:40.683097 11966 raft_consensus.cc:2463] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 8.
I20250113 08:59:40.659373 15273 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.691275 12006 leader_election.cc:400] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539) with higher term. Message: Invalid argument: T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7.
I20250113 08:59:40.675412 11893 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:40.695422 12007 leader_election.cc:304] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:40.696177 11966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.702528 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.680424 15323 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:40.709765 11932 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:40.711249 11932 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:40.712126 11859 leader_election.cc:336] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 3.152s (SENT)
I20250113 08:59:40.712625 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.618429 15031 consensus_queue.cc:260] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.728044 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.706497 15179 raft_consensus.cc:2798] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election won for term 8
W20250113 08:59:40.735749 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 65709.1us
I20250113 08:59:40.734611 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.726123 15273 raft_consensus.cc:2798] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:40.731101 15341 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:40.694051 15059 raft_consensus.cc:513] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.745435 15273 raft_consensus.cc:695] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:40.746237 15059 leader_election.cc:290] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.746613 15273 consensus_queue.cc:237] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.748345 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:40.748888 15309 leader_election.cc:290] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.740602 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.743157 12006 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.752761 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:40.754755 11932 leader_election.cc:304] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:40.759931 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.739537 15179 raft_consensus.cc:695] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:40.752390 15322 consensus_meta.cc:287] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Time spent flushing consensus metadata: real 0.512s user 0.012s sys 0.000s
I20250113 08:59:40.759258 15323 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:40.769598 15231 raft_consensus.cc:3049] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:40.774875 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 1997) took 2276 ms. Trace:
I20250113 08:59:40.809599 11892 rpcz_store.cc:276] 0113 08:59:38.498204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.180725 (+1682521us) service_pool.cc:225] Handling call
0113 08:59:40.774686 (+593961us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b54003af054. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.774819 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27747712}
I20250113 08:59:40.780789 11891 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:40.783375 15065 raft_consensus.cc:3054] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:59:40.785117 14879 raft_consensus.cc:2743] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:40.786295 12040 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:40.801991 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.814473 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:40.811302 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 81146.8us
I20250113 08:59:40.788923 15021 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:40.823712 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 84133us
I20250113 08:59:40.780671 15344 raft_consensus.cc:2743] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:40.832201 11893 raft_consensus.cc:1235] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250113 08:59:40.828998 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:40.830709 12006 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:40.838584 14664 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.840087 11966 raft_consensus.cc:2388] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:40.843225 15204 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:40.844596 11888 raft_consensus.cc:2388] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 8: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:40.849759 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:40.850637 15342 log.cc:926] Time spent T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20250113 08:59:40.850919 11893 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.853188 11892 raft_consensus.cc:2405] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250113 08:59:40.809509 15231 raft_consensus.cc:738] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:40.854633 15231 consensus_queue.cc:260] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.855857 15231 raft_consensus.cc:3054] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:40.855573 11892 pending_rounds.cc:77] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 5
I20250113 08:59:40.856557 11892 pending_rounds.cc:99] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20250113 08:59:40.856842 11892 raft_consensus.cc:2883] T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:40.857767 15070 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:40.865604 11885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 1991) took 9837 ms. Trace:
I20250113 08:59:40.811340 15311 raft_consensus.cc:2743] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:59:40.866052 11885 rpcz_store.cc:276] 0113 08:59:30.994763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:31.065541 (+ 70778us) service_pool.cc:225] Handling call
0113 08:59:31.085100 (+ 19559us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:38.213795 (+7128695us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.831982 (+2618187us) spinlock_profiling.cc:243] Waited 6.3 s on lock 0x7b540015fed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:40.832158 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14492071680}
W20250113 08:59:40.879550 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 48977us
I20250113 08:59:40.885504 11893 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:40.887367 11932 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:40.888242 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:40.888990 12042 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.841117 14664 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.817749 15303 raft_consensus.cc:491] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:40.898747 11891 raft_consensus.cc:1235] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:40.899525 11960 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:40.900278 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2301) took 7379 ms. Trace:
I20250113 08:59:40.900583 11960 rpcz_store.cc:276] 0113 08:59:33.521014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.876222 (+7355208us) service_pool.cc:225] Handling call
0113 08:59:40.899249 (+ 23027us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.900042 (+ 793us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.900229 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:40.800856 15322 raft_consensus.cc:513] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.892536 15224 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:40.920018 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.920466 11960 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:40.913007 15204 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:40.915035 14879 consensus_queue.cc:1046] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20250113 08:59:40.868335 15070 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.924818 14865 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:40.929157 14879 consensus_queue.cc:1225] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 7.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250113 08:59:40.931543 14865 raft_consensus.cc:695] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:40.932385 15247 raft_consensus.cc:513] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.911850 15303 raft_consensus.cc:513] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.937762 15314 raft_consensus.cc:491] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:40.938427 15314 raft_consensus.cc:513] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.939589 15247 leader_election.cc:290] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.940269 11889 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:40.940367 15279 raft_consensus.cc:3049] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:40.940699 15058 raft_consensus.cc:2743] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:40.940882 15279 raft_consensus.cc:738] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:40.941872 15279 consensus_queue.cc:260] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.943042 15279 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:40.942508 11889 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:40.894359 15179 consensus_queue.cc:237] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:40.945644 11962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2285) took 5224 ms (client timeout 1976 ms). Trace:
W20250113 08:59:40.946099 11962 rpcz_store.cc:269] 0113 08:59:35.721004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.511664 (+1790660us) service_pool.cc:225] Handling call
0113 08:59:40.945463 (+3433799us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540028e1d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.945572 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":555162880}
I20250113 08:59:40.947196 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:40.947788 11962 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:40.952482 11959 raft_consensus.cc:2463] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:40.953828 12042 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
W20250113 08:59:40.954473 11963 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 547) took 5685 ms (client timeout 2531 ms). Trace:
W20250113 08:59:40.955080 11963 rpcz_store.cc:269] 0113 08:59:35.269049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.678128 (+2409079us) service_pool.cc:225] Handling call
0113 08:59:40.954175 (+3276047us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b540006e0d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.954379 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1153607552}
I20250113 08:59:40.957731 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:40.919624 15224 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:40.969467 15224 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:40.971736 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:40.938427 15322 leader_election.cc:290] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Requested vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:40.975589 14783 raft_consensus.cc:3049] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:40.976225 14783 raft_consensus.cc:738] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:40.977416 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2302) took 7456 ms. Trace:
W20250113 08:59:40.964864 11961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54924 (request call id 532) took 5805 ms (client timeout 1504 ms). Trace:
W20250113 08:59:40.978214 11961 rpcz_store.cc:269] 0113 08:59:35.159247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:35.167966 (+ 8719us) service_pool.cc:225] Handling call
0113 08:59:40.953773 (+5785807us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b540015d454. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:40.964796 (+ 11023us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3907873024}
W20250113 08:59:40.967572 15265 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 2.656s user 0.012s sys 0.019s
I20250113 08:59:40.989794 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2332) took 3343 ms. Trace:
I20250113 08:59:40.981391 14783 consensus_queue.cc:260] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:40.982046 15282 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:40.985978 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:41.002288 11961 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
W20250113 08:59:40.977461 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 97216.1us
I20250113 08:59:40.990850 11960 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:40.991457 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:41.006263 12042 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:40.990192 12034 rpcz_store.cc:276] 0113 08:59:37.645821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.647330 (+ 1509us) service_pool.cc:225] Handling call
0113 08:59:37.647803 (+ 473us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:37.977352 (+329549us) raft_consensus.cc:1529] Early marking committed up to index 2
0113 08:59:37.977425 (+ 73us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:38.064034 (+ 86609us) log.cc:844] Serialized 26 byte log entry
0113 08:59:38.282888 (+218854us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:38.282937 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:38.283034 (+ 97us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:40.967017 (+2683983us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:40.989318 (+ 22301us) raft_consensus.cc:1680] finished
0113 08:59:40.989344 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.989595 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":334222,"raft.run_cpu_time_us":646,"raft.run_wall_time_us":76071,"spinlock_wait_cycles":12721024,"thread_start_us":184597,"threads_started":1,"wal-append.queue_time_us":230598}
I20250113 08:59:40.994472 14956 raft_consensus.cc:2743] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:59:41.017468 12039 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20250113 08:59:41.017808 12037 raft_consensus.cc:1235] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:41.019373 15045 raft_consensus.cc:3054] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:41.019852 11884 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:40.995985 15070 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.021797 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:41.022406 11932 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:41.023397 11884 raft_consensus.cc:1270] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250113 08:59:41.023836 11884 pending_rounds.cc:77] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 2
I20250113 08:59:41.024117 11884 pending_rounds.cc:99] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250113 08:59:41.024403 11884 raft_consensus.cc:2883] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:40.977854 11964 rpcz_store.cc:276] 0113 08:59:33.521360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.976451 (+7455091us) service_pool.cc:225] Handling call
0113 08:59:40.976840 (+ 389us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.977049 (+ 209us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:40.977086 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:40.977129 (+ 43us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:40.977162 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.977215 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:40.977331 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:41.007740 11931 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:41.007884 11858 leader_election.cc:304] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:41.016180 12006 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:41.016170 15330 log.cc:926] Time spent T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.482s user 0.001s sys 0.000s
I20250113 08:59:41.037611 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.003206 15282 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.045854 12042 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:41.050114 12038 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250113 08:59:41.056380 11856 leader_election.cc:304] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:41.059026 11960 raft_consensus.cc:1235] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:41.060165 11884 raft_consensus.cc:1270] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250113 08:59:41.061976 12038 raft_consensus.cc:1270] T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250113 08:59:41.038072 15058 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:41.045413 15294 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:41.047541 15053 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:40.932533 14865 consensus_queue.cc:237] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:41.067914 15179 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250113 08:59:41.066254 15247 raft_consensus.cc:2798] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:41.047823 15322 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
I20250113 08:59:41.089177 15107 consensus_queue.cc:1035] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20250113 08:59:41.086011 15129 raft_consensus.cc:2798] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:59:41.076002 15294 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:41.094487 15294 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:41.094398 15129 raft_consensus.cc:695] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [term 8 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:41.079612 15300 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
W20250113 08:59:41.119807 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 112540us
I20250113 08:59:41.098031 15129 consensus_queue.cc:237] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:41.070037 15344 consensus_queue.cc:1035] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20250113 08:59:41.114343 11962 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 9.
I20250113 08:59:41.080853 15053 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.133939 12007 leader_election.cc:304] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:41.106923 15309 consensus_queue.cc:1035] T 54d2a9b9e9134754acb8568b23bba884 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20250113 08:59:41.142525 11890 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:41.121599 15247 raft_consensus.cc:695] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:41.149366 11888 raft_consensus.cc:1270] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250113 08:59:41.149325 15179 raft_consensus.cc:2798] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader election won for term 9
W20250113 08:59:41.219446 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 98732.6us
I20250113 08:59:41.227974 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.228317 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.228726 12717 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:41.152387 15179 raft_consensus.cc:695] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
W20250113 08:59:41.240779 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 2.28127e+06us
I20250113 08:59:41.247591 12724 raft_consensus.cc:2405] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 9 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250113 08:59:41.224115 15256 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:41.248236 15353 log.cc:926] Time spent T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20250113 08:59:41.263098 14994 consensus_queue.cc:1046] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20250113 08:59:41.274073 15268 raft_consensus.cc:491] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:41.274749 15053 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.274760 15268 raft_consensus.cc:513] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.253847 15256 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.279712 15256 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:41.276242 15190 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:41.281131 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 703) took 2553 ms. Trace:
I20250113 08:59:41.281077 15190 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.281611 12720 rpcz_store.cc:276] 0113 08:59:38.727615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.929274 (+201659us) service_pool.cc:225] Handling call
0113 08:59:38.961362 (+ 32088us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:38.986812 (+ 25450us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:38.990759 (+ 3947us) log.cc:844] Serialized 14 byte log entry
0113 08:59:41.280714 (+2289955us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:41.280843 (+ 129us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:41.280891 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.280967 (+ 76us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:41.281076 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2122955,"prepare.queue_time_us":976,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":20412,"thread_start_us":166510,"threads_started":1,"wal-append.queue_time_us":408085,"wal-append.run_cpu_time_us":1187,"wal-append.run_wall_time_us":1001440}
I20250113 08:59:41.283248 15190 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:41.299748 12006 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:41.306906 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.295369 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 836) took 2670 ms. Trace:
W20250113 08:59:41.289819 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 2.08342e+06us
I20250113 08:59:41.310830 12725 rpcz_store.cc:276] 0113 08:59:38.624790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.879097 (+254307us) service_pool.cc:225] Handling call
0113 08:59:40.174358 (+1295261us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:40.190841 (+ 16483us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.295191 (+1104350us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54006c9650. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.295310 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2895262080}
I20250113 08:59:41.290328 12721 raft_consensus.cc:1270] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250113 08:59:41.314560 11859 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:41.310850 12724 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 5.
I20250113 08:59:41.283627 15268 leader_election.cc:290] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.302773 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dda9aa8769e4146a4dabeadd588c734" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.246273 15261 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:41.295472 12686 connection.cc:232] RPC call timeout handler was delayed by -2.7146802769999567s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.36850000000000005s for a response
I20250113 08:59:41.334823 12722 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:41.307981 15338 log.cc:926] Time spent T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20250113 08:59:41.284583 15355 log.cc:926] Time spent T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
W20250113 08:59:41.363039 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 92968.5us
I20250113 08:59:41.366055 12006 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:41.338289 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57686 (request call id 652) took 1044 ms. Trace:
I20250113 08:59:41.373773 11893 rpcz_store.cc:276] 0113 08:59:40.293902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.894465 (+600563us) service_pool.cc:225] Handling call
0113 08:59:40.914320 (+ 19855us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.914607 (+ 287us) raft_consensus.cc:1529] Early marking committed up to index 7
0113 08:59:41.103243 (+188636us) log.cc:844] Serialized 14 byte log entry
0113 08:59:41.337098 (+233855us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:41.337300 (+ 202us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:41.337527 (+ 227us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.337940 (+ 413us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:41.338221 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7152,"thread_start_us":343917,"threads_started":2}
I20250113 08:59:41.369027 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.386618 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:41.386955 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.385135 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.387980 12724 raft_consensus.cc:2463] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:41.388190 12042 raft_consensus.cc:2463] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:41.390271 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.391029 12720 raft_consensus.cc:3054] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:41.387641 12721 raft_consensus.cc:2463] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:41.285495 14994 raft_consensus.cc:3049] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:41.401583 12725 raft_consensus.cc:2388] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:41.328995 15231 raft_consensus.cc:2743] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250113 08:59:41.404304 12686 leader_election.cc:336] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 2.787s (SENT)
W20250113 08:59:41.415253 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 49302.6us
I20250113 08:59:41.416276 11893 pending_rounds.cc:77] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 3
I20250113 08:59:41.422787 11893 pending_rounds.cc:99] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250113 08:59:41.423159 11893 raft_consensus.cc:2883] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:41.423203 15150 raft_consensus.cc:3049] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:41.423785 15150 raft_consensus.cc:738] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:41.403951 14994 raft_consensus.cc:738] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
W20250113 08:59:41.426239 12719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 905) took 3841 ms (client timeout 3152 ms). Trace:
I20250113 08:59:41.426265 14994 consensus_queue.cc:260] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:41.426774 12719 rpcz_store.cc:269] 0113 08:59:37.584985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:37.676044 (+ 91059us) service_pool.cc:225] Handling call
0113 08:59:41.425979 (+3749935us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54006211d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.426166 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":678930176}
I20250113 08:59:41.427536 14994 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:41.428089 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:41.434134 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 49147.5us
I20250113 08:59:41.414778 12686 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:41.428256 12684 connection.cc:232] RPC call timeout handler was delayed by -1.2878915579999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
I20250113 08:59:41.429539 15054 raft_consensus.cc:491] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:41.428999 15189 raft_consensus.cc:491] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:41.429755 15049 raft_consensus.cc:491] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:41.429962 14966 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:41.430142 11888 pending_rounds.cc:77] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 6
I20250113 08:59:41.385526 15313 raft_consensus.cc:2743] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:41.426321 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 695) took 4079 ms. Trace:
W20250113 08:59:41.499018 12684 connection.cc:232] RPC call timeout handler was delayed by -1.1429915579999488s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
I20250113 08:59:41.507632 12718 rpcz_store.cc:276] 0113 08:59:37.347375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.240243 (+892868us) service_pool.cc:225] Handling call
0113 08:59:38.822715 (+582472us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:38.882333 (+ 59618us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.426089 (+2543756us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b5400501250. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.426253 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1303208320}
W20250113 08:59:41.508080 12684 connection.cc:232] RPC call timeout handler was delayed by -1.1231915580000305s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
I20250113 08:59:41.502363 11888 pending_rounds.cc:99] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20250113 08:59:41.508507 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:41.508903 12684 connection.cc:232] RPC call timeout handler was delayed by -1.0853915579999693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
I20250113 08:59:41.421841 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2350) took 1952 ms. Trace:
W20250113 08:59:41.509277 12684 connection.cc:232] RPC call timeout handler was delayed by -1.0790915579999592s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
I20250113 08:59:41.509006 11888 raft_consensus.cc:2883] T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:41.501456 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.291689 15224 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.509589 12724 raft_consensus.cc:2388] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 7: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:41.509413 12035 rpcz_store.cc:276] 0113 08:59:39.468954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.217957 (+749003us) service_pool.cc:225] Handling call
0113 08:59:40.218266 (+ 309us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.234749 (+ 16483us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.421576 (+1186827us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b54002b0554. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:41.421775 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30621952}
I20250113 08:59:41.510288 12721 raft_consensus.cc:3054] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:41.517019 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:41.520956 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 265645us
W20250113 08:59:41.521526 12685 connection.cc:232] RPC call timeout handler was delayed by -1.1111587080000618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
I20250113 08:59:41.399720 14879 raft_consensus.cc:2743] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:41.534767 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.534963 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.535799 12719 raft_consensus.cc:2405] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250113 08:59:41.405216 15309 raft_consensus.cc:491] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:41.541235 12726 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250113 08:59:41.541700 12726 pending_rounds.cc:77] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting all ops after (but not including) 7
I20250113 08:59:41.541978 12726 pending_rounds.cc:99] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250113 08:59:41.542259 12726 raft_consensus.cc:2883] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:41.528208 12006 connection.cc:232] RPC call timeout handler was delayed by -1.6162097139999787s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
I20250113 08:59:41.296415 15247 consensus_queue.cc:237] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.307720 15179 consensus_queue.cc:237] T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:41.544255 12006 connection.cc:232] RPC call timeout handler was delayed by -1.3336097139999765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
W20250113 08:59:41.536809 14956 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.046s sys 0.088s
I20250113 08:59:41.299062 15294 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.424728 15150 consensus_queue.cc:260] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:41.563165 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:41.545396 15265 log.cc:926] Time spent T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20250113 08:59:41.575932 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.536173 15053 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:41.536427 15231 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:41.570120 14956 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.551s user 0.046s sys 0.089s
I20250113 08:59:41.520352 14790 consensus_queue.cc:1035] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.052s
W20250113 08:59:41.610670 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 225462us
I20250113 08:59:41.611212 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2266) took 8621 ms. Trace:
W20250113 08:59:41.611199 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 379439us
I20250113 08:59:41.587532 15302 consensus_queue.cc:1035] T 08aa17121f5b429b8a32299911c7ef76 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20250113 08:59:41.625713 11887 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250113 08:59:41.519461 14966 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.592197 15150 raft_consensus.cc:3054] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:41.614935 11965 rpcz_store.cc:276] 0113 08:59:32.970723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:34.800350 (+1829627us) service_pool.cc:225] Handling call
0113 08:59:37.133090 (+2332740us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:38.355466 (+1222376us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.381973 (+2026507us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540006ead4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:41.591489 (+1209516us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b540006ead0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.591641 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7916854272}
W20250113 08:59:41.624598 11890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 2012) took 2997 ms (client timeout 1957 ms). Trace:
I20250113 08:59:41.538813 15065 consensus_queue.cc:1046] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.052s
I20250113 08:59:41.534552 15054 raft_consensus.cc:513] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.642503 14879 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:41.633692 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.712623 12719 raft_consensus.cc:2405] T 07a14a3957e44925ae0925b645d220eb P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250113 08:59:41.654666 11968 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2301) took 2129 ms (client timeout 1734 ms). Trace:
W20250113 08:59:41.713256 11968 rpcz_store.cc:269] 0113 08:59:39.525394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.890075 (+364681us) service_pool.cc:225] Handling call
0113 08:59:41.654370 (+1764295us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54002a4c54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.654571 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":241091328}
I20250113 08:59:41.714509 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.519536 15189 raft_consensus.cc:513] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.519517 15049 raft_consensus.cc:513] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.625346 12684 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:41.619923 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
W20250113 08:59:41.651346 12686 connection.cc:232] RPC call timeout handler was delayed by -1.6229189359999054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
I20250113 08:59:41.718171 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:41.722311 12686 connection.cc:232] RPC call timeout handler was delayed by -1.5839127059999782s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31200000000000006s for a response
W20250113 08:59:41.722784 12686 connection.cc:232] RPC call timeout handler was delayed by -1.4771189359998971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
W20250113 08:59:41.723148 12686 connection.cc:232] RPC call timeout handler was delayed by -1.4204189359999191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
W20250113 08:59:41.723465 12686 connection.cc:232] RPC call timeout handler was delayed by -1.0874189359999491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.211s for a response
W20250113 08:59:41.723804 12686 connection.cc:232] RPC call timeout handler was delayed by -1.0865189359999476s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20250113 08:59:41.722805 11968 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 8.
I20250113 08:59:41.552139 15309 raft_consensus.cc:513] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:41.687971 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 152623us
I20250113 08:59:41.629482 14790 consensus_queue.cc:1225] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250113 08:59:41.591748 15231 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:41.640061 15303 leader_election.cc:290] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.591547 15053 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:41.699915 12006 leader_election.cc:336] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.643s (SENT)
W20250113 08:59:41.699786 11890 rpcz_store.cc:269] 0113 08:59:38.627116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.386622 (+759506us) service_pool.cc:225] Handling call
0113 08:59:41.624364 (+2237742us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54000d7d54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.624506 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3922322816}
I20250113 08:59:41.751849 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.703336 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1912) took 1258 ms. Trace:
I20250113 08:59:41.703152 12685 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
W20250113 08:59:41.785933 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 82862.3us
I20250113 08:59:41.787595 12685 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:41.784488 11889 rpcz_store.cc:276] 0113 08:59:40.445053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.946143 (+501090us) service_pool.cc:225] Handling call
0113 08:59:40.946441 (+ 298us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:40.947525 (+ 1084us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:40.947557 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:40.948025 (+ 468us) log.cc:844] Serialized 26 byte log entry
0113 08:59:41.325746 (+377721us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:41.702054 (+376308us) log.cc:844] Serialized 14 byte log entry
0113 08:59:41.702949 (+ 895us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.703048 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:41.703097 (+ 49us) raft_consensus.cc:1680] finished
0113 08:59:41.703114 (+ 17us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:41.703267 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":297464,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":31,"raft.queue_time_us":554,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":12013,"thread_start_us":685718,"threads_started":2,"wal-append.queue_time_us":472632}
I20250113 08:59:41.792505 12033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1808) took 1540 ms. Trace:
W20250113 08:59:41.792732 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33878 (request call id 1776) took 3077 ms (client timeout 3776 ms). Trace:
W20250113 08:59:41.793118 12040 rpcz_store.cc:269] 0113 08:59:38.715209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.723323 (+ 8114us) service_pool.cc:225] Handling call
0113 08:59:41.792581 (+3069258us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b540024f9d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.792678 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4670250240}
I20250113 08:59:41.793641 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2307) took 1604 ms. Trace:
I20250113 08:59:41.794026 11959 rpcz_store.cc:276] 0113 08:59:40.189361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.608972 (+419611us) service_pool.cc:225] Handling call
0113 08:59:41.793495 (+1184523us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54001bdfd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:41.793586 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":753512832}
I20250113 08:59:41.794953 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc31099ea8040b7bfbfe7d779148ca5" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.795300 12040 raft_consensus.cc:1235] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:41.795743 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2274) took 7154 ms. Trace:
I20250113 08:59:41.796166 14966 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.797211 14981 consensus_queue.cc:1046] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:41.797892 11858 leader_election.cc:304] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:41.798122 14981 raft_consensus.cc:3049] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:41.798602 14981 raft_consensus.cc:738] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:41.799430 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1928) took 1108 ms. Trace:
I20250113 08:59:41.687268 15251 raft_consensus.cc:491] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:41.729605 11934 leader_election.cc:336] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 2.199s (SENT)
I20250113 08:59:41.800303 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:41.800541 15251 raft_consensus.cc:513] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:41.800940 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 71403.7us
I20250113 08:59:41.801728 12724 raft_consensus.cc:2463] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:41.735358 12686 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:41.736725 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.725198 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bbae0b61a948339e4c443b9237ad87" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:41.803310 11968 raft_consensus.cc:3054] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:41.747378 12725 raft_consensus.cc:2388] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a in current term 6: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:41.774295 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:41.803781 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.804422 12719 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:41.804750 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:41.805588 11890 raft_consensus.cc:2405] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250113 08:59:41.729000 11965 raft_consensus.cc:2363] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8.
I20250113 08:59:41.806355 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:41.807024 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 71747.2us
I20250113 08:59:41.779455 15007 raft_consensus.cc:2743] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:41.808027 12686 leader_election.cc:304] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:41.808208 15031 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:41.808701 15007 raft_consensus.cc:2743] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:41.792927 12033 rpcz_store.cc:276] 0113 08:59:40.252579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.371817 (+119238us) service_pool.cc:225] Handling call
0113 08:59:40.384700 (+ 12883us) raft_consensus.cc:1466] Updating replica for 4 ops
0113 08:59:40.393373 (+ 8673us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:40.393402 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 4 ops
0113 08:59:40.401750 (+ 8348us) log.cc:844] Serialized 104 byte log entry
0113 08:59:40.506303 (+104553us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:40.506342 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.506414 (+ 72us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:41.790876 (+1284462us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b24001e6248. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333528264 00007fc33352636b 00007fc3335b4b19 00007fc3335b47a5 00007fc3335b3d00 00007fc3362b3f88 00007fc3362b46e8 00007fc33628bf98 00007fc3362873aa 00007fc3362867f6
0113 08:59:41.792187 (+ 1311us) raft_consensus.cc:1680] finished
0113 08:59:41.792205 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:41.792445 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26495,"raft.queue_time_us":16154,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":16784,"spinlock_wait_cycles":51053312,"thread_start_us":55808,"threads_started":1,"wal-append.queue_time_us":140614}
I20250113 08:59:41.799861 11893 rpcz_store.cc:276] 0113 08:59:40.690819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.375079 (+684260us) service_pool.cc:225] Handling call
0113 08:59:41.416045 (+ 40966us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:41.423998 (+ 7953us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:41.424043 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:41.424602 (+ 559us) log.cc:844] Serialized 26 byte log entry
0113 08:59:41.688653 (+264051us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:41.688796 (+ 143us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.689086 (+ 290us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:41.799215 (+110129us) raft_consensus.cc:1680] finished
0113 08:59:41.799236 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:41.799370 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":263249,"threads_started":1,"wal-append.queue_time_us":345362}
I20250113 08:59:41.789012 15190 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:41.721457 14879 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.816793 11968 raft_consensus.cc:2463] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
I20250113 08:59:41.820022 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.820101 12686 leader_election.cc:304] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:41.800194 12041 raft_consensus.cc:1212] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-7.5] Dedup: 3.3->[5.4-7.5]
I20250113 08:59:41.823148 12041 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20250113 08:59:41.807415 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:41.824067 11890 raft_consensus.cc:3054] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:41.806973 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.825373 11965 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:41.796077 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.825938 11959 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:41.806401 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:41.812021 12717 consensus_meta.cc:287] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.563s user 0.010s sys 0.000s
I20250113 08:59:41.803004 14664 consensus_queue.cc:1035] T 67a8b82ef293439e815f49903f8e5dbb P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250113 08:59:41.828315 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 137883us
W20250113 08:59:41.828729 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 203519us
W20250113 08:59:41.829514 12006 leader_election.cc:336] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539): Timed out: RequestConsensusVote RPC to 127.10.115.129:33539 timed out after 1.957s (SENT)
I20250113 08:59:41.812616 12722 raft_consensus.cc:2463] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 8.
I20250113 08:59:41.790973 15041 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:41.796083 11967 rpcz_store.cc:276] 0113 08:59:34.641558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:39.377174 (+4735616us) service_pool.cc:225] Handling call
0113 08:59:39.436646 (+ 59472us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.550413 (+1113767us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.795550 (+1245137us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540029f754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:41.795685 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2559469824}
I20250113 08:59:41.797077 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2023) took 1882 ms. Trace:
I20250113 08:59:41.800976 11889 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:41.802803 11934 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:41.834362 11891 rpcz_store.cc:276] 0113 08:59:39.915152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.833708 (+918556us) service_pool.cc:225] Handling call
0113 08:59:40.877303 (+ 43595us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:40.899184 (+ 21881us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.796850 (+897666us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b540038f254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:41.797006 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48848768}
I20250113 08:59:41.827102 12717 raft_consensus.cc:2463] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 8.
I20250113 08:59:41.802862 15311 raft_consensus.cc:2798] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:41.831459 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.831188 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a8b82ef293439e815f49903f8e5dbb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.832742 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43de3c8f06b48358bcee58f9567bb3b" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.816457 15190 raft_consensus.cc:491] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:41.840999 12722 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:41.841888 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d70ffc378340e785c4f8dde909f9ba" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:41.842629 11959 raft_consensus.cc:2463] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 5.
I20250113 08:59:41.836586 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.802771 15251 leader_election.cc:290] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:41.819525 15224 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.844544 12717 raft_consensus.cc:2371] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7. Re-sending same reply.
I20250113 08:59:41.851893 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.827386 11858 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:41.850718 12006 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:41.858068 12684 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:41.828081 15007 raft_consensus.cc:2798] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:41.852772 12719 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:41.861408 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.862784 12684 leader_election.cc:304] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:41.841615 14865 raft_consensus.cc:2743] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250113 08:59:41.848044 15294 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.848560 15311 raft_consensus.cc:695] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Leader. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:41.799422 14981 consensus_queue.cc:260] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.861768 15256 raft_consensus.cc:2798] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:41.866451 15065 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250113 08:59:41.870085 14966 raft_consensus.cc:2743] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250113 08:59:41.873229 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 53210.4us
I20250113 08:59:41.891012 12686 leader_election.cc:304] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:41.844026 15190 raft_consensus.cc:3054] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:41.876427 12033 raft_consensus.cc:1270] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250113 08:59:41.877283 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.871009 12721 raft_consensus.cc:2463] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:41.879580 11959 raft_consensus.cc:1235] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [6.8-7.9]
I20250113 08:59:41.886021 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.873263 11967 raft_consensus.cc:1235] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:41.873900 14994 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:41.879487 15302 consensus_queue.cc:1046] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20250113 08:59:41.966226 11965 raft_consensus.cc:1235] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:41.992307 11889 raft_consensus.cc:2463] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:41.965880 12686 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:41.969959 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.879499 12038 raft_consensus.cc:1270] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250113 08:59:42.027695 12722 raft_consensus.cc:3054] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:42.027817 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:41.887122 12720 raft_consensus.cc:2463] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 6.
I20250113 08:59:42.029819 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2283) took 6781 ms. Trace:
I20250113 08:59:42.030289 11959 rpcz_store.cc:276] 0113 08:59:35.248887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.843326 (+6594439us) service_pool.cc:225] Handling call
0113 08:59:41.879251 (+ 35925us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:42.029481 (+150230us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.029748 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:41.897207 11890 raft_consensus.cc:2463] T cee55da481c64a6d885aa3ab442761cd P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 9.
I20250113 08:59:42.032845 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a40dc5e61345558c943df4be928468" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:42.025899 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2281) took 6812 ms. Trace:
I20250113 08:59:42.037036 11967 rpcz_store.cc:276] 0113 08:59:35.213986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.843216 (+6629230us) service_pool.cc:225] Handling call
0113 08:59:41.873068 (+ 29852us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.025546 (+152478us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.025821 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.038545 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2311) took 6772 ms. Trace:
I20250113 08:59:42.048552 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:41.882894 15256 raft_consensus.cc:491] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:41.878826 14798 raft_consensus.cc:2743] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:41.860697 15007 raft_consensus.cc:695] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:41.887359 15341 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:42.042470 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8133c246c9d74113aec7a002df391ef7" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:42.084910 12723 raft_consensus.cc:2405] T 8133c246c9d74113aec7a002df391ef7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250113 08:59:42.084925 12033 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:42.050835 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:42.045459 11965 rpcz_store.cc:276] 0113 08:59:35.266615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.861080 (+6594465us) service_pool.cc:225] Handling call
0113 08:59:41.898901 (+ 37821us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.037980 (+139079us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.038479 (+ 499us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.040191 15261 raft_consensus.cc:2743] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:42.025440 14994 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:41.884099 15065 raft_consensus.cc:491] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:42.029121 11889 raft_consensus.cc:2405] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20250113 08:59:41.873625 15357 log.cc:926] Time spent T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20250113 08:59:42.106343 11959 raft_consensus.cc:1235] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 9. Ops: []
I20250113 08:59:42.113723 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2284) took 6787 ms. Trace:
I20250113 08:59:42.114104 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:42.114202 11959 rpcz_store.cc:276] 0113 08:59:35.326378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.031848 (+6705470us) service_pool.cc:225] Handling call
0113 08:59:42.105995 (+ 74147us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.113389 (+ 7394us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.113604 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.040674 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aa379773c047589ae2362850997a47" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.883042 14981 raft_consensus.cc:3054] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:41.869544 15129 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:42.125768 11889 raft_consensus.cc:1235] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:42.126498 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1935) took 1225 ms. Trace:
I20250113 08:59:42.126919 11889 rpcz_store.cc:276] 0113 08:59:40.900770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.107715 (+1206945us) service_pool.cc:225] Handling call
0113 08:59:42.120462 (+ 12747us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.126248 (+ 5786us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.126427 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.104787 14994 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:42.107992 15065 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:42.093621 15214 raft_consensus.cc:2798] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250113 08:59:42.094033 15054 leader_election.cc:290] T 8b1c553832b94166a0a6089104a74f6c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.055055 15031 consensus_queue.cc:1035] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20250113 08:59:42.094982 14879 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:41.882565 15311 consensus_queue.cc:237] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:41.880089 15309 leader_election.cc:290] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.095587 15341 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:42.078450 15256 raft_consensus.cc:3054] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:42.096691 14798 raft_consensus.cc:2743] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:41.880175 15273 consensus_queue.cc:1035] T 9a457bc5206c46d7b198dbff2b1c137c P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.217s
I20250113 08:59:42.097673 11967 raft_consensus.cc:1212] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-5.4] Dedup: 5.4->[]
I20250113 08:59:42.185045 11967 raft_consensus.cc:1235] T e2677547aaee4bba86cf339fe13278bf P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20250113 08:59:42.185993 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2286) took 6345 ms. Trace:
I20250113 08:59:42.186455 11967 rpcz_store.cc:276] 0113 08:59:35.840539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.052584 (+6212045us) service_pool.cc:225] Handling call
0113 08:59:42.097550 (+ 44966us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:42.185642 (+ 88092us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.185891 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.188213 11967 raft_consensus.cc:1235] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 8. Ops: []
I20250113 08:59:42.098017 11965 raft_consensus.cc:1270] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250113 08:59:42.098575 12719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:41.884627 15231 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:42.052749 15359 log.cc:926] Time spent T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.372s user 0.001s sys 0.000s
I20250113 08:59:42.101887 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218ea9985e1d40a48892a8e01aaaef39" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:42.102079 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2276) took 7375 ms. Trace:
I20250113 08:59:42.017699 15053 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.032385 15268 raft_consensus.cc:491] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:42.017943 15314 leader_election.cc:290] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.003506 15189 leader_election.cc:290] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.119336 12720 raft_consensus.cc:2463] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:42.119822 12723 raft_consensus.cc:2405] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250113 08:59:42.120962 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2361) took 1725 ms. Trace:
I20250113 08:59:42.122875 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43888 (request call id 2309) took 1907 ms. Trace:
I20250113 08:59:42.221391 11966 rpcz_store.cc:276] 0113 08:59:40.215628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.695935 (+480307us) service_pool.cc:225] Handling call
0113 08:59:42.122571 (+1426636us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54001ad1d4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:42.122785 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":317811968}
I20250113 08:59:42.223119 11966 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:41.880299 15070 consensus_queue.cc:1035] T b300f59ff3964ee28bb6ef0c1141395e P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.203s
I20250113 08:59:41.879179 15049 leader_election.cc:290] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:42.125974 15129 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:42.148658 15007 consensus_queue.cc:237] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.159534 12033 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20250113 08:59:42.235006 12033 pending_rounds.cc:77] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 5
I20250113 08:59:42.235054 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2290) took 4505 ms. Trace:
I20250113 08:59:42.235682 12033 pending_rounds.cc:99] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.6
I20250113 08:59:42.235793 11966 rpcz_store.cc:276] 0113 08:59:37.723655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.222495 (+4498840us) service_pool.cc:225] Handling call
0113 08:59:42.222901 (+ 406us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.228312 (+ 5411us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.228541 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.237929 11966 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250113 08:59:42.145404 12722 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
W20250113 08:59:42.135625 15375 log.cc:926] Time spent T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20250113 08:59:42.180423 11959 raft_consensus.cc:3054] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:42.189105 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2288) took 6101 ms. Trace:
I20250113 08:59:42.164384 15214 raft_consensus.cc:491] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:42.198550 11961 rpcz_store.cc:276] 0113 08:59:34.726388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.003073 (+6276685us) service_pool.cc:225] Handling call
0113 08:59:41.006085 (+ 3012us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.015638 (+ 9553us) raft_consensus.cc:1529] Early marking committed up to index 9
0113 08:59:41.015977 (+ 339us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.101506 (+1085529us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.101710 (+ 204us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:42.101773 (+ 63us) raft_consensus.cc:1640] Marking committed up to 9
0113 08:59:42.101820 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.101880 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.102026 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":755542,"prepare.queue_time_us":758455,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":3486,"raft.queue_time_us":1143,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":47961,"spinlock_wait_cycles":4502784,"thread_start_us":328156,"threads_started":1,"wal-append.queue_time_us":557503}
I20250113 08:59:42.221194 12041 rpcz_store.cc:276] 0113 08:59:40.395649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.743975 (+348326us) service_pool.cc:225] Handling call
0113 08:59:41.799901 (+1055926us) raft_consensus.cc:1466] Updating replica for 3 ops
0113 08:59:41.832674 (+ 32773us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.120680 (+288006us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001f18d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:42.120884 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2427497728}
I20250113 08:59:42.225792 12723 raft_consensus.cc:1235] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250113 08:59:42.221457 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2317) took 4688 ms. Trace:
I20250113 08:59:42.243520 11967 rpcz_store.cc:276] 0113 08:59:37.515988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.187539 (+4671551us) service_pool.cc:225] Handling call
0113 08:59:42.188016 (+ 477us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.194923 (+ 6907us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.201570 (+ 6647us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.242739 11965 rpcz_store.cc:276] 0113 08:59:36.087983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.086878 (+5998895us) service_pool.cc:225] Handling call
0113 08:59:42.097632 (+ 10754us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.188785 (+ 91153us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.189038 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.249402 11967 raft_consensus.cc:1235] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:42.250285 11966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 548) took 2985 ms. Trace:
I20250113 08:59:42.250808 11966 rpcz_store.cc:276] 0113 08:59:39.265410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.237136 (+2971726us) service_pool.cc:225] Handling call
0113 08:59:42.237573 (+ 437us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.246059 (+ 8486us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.250218 (+ 4159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.252888 12722 raft_consensus.cc:3049] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 LEADER]: Stepping down as leader of term 9
I20250113 08:59:42.253657 12722 raft_consensus.cc:738] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:42.236241 12033 raft_consensus.cc:2911] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Aborting config change with OpId 6.6: Aborted: Op aborted by new leader
I20250113 08:59:42.254803 12033 raft_consensus.cc:2883] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:42.257197 11965 raft_consensus.cc:1235] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250113 08:59:42.257779 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 550) took 2882 ms. Trace:
I20250113 08:59:42.258252 11967 rpcz_store.cc:276] 0113 08:59:39.367986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.248948 (+2880962us) service_pool.cc:225] Handling call
0113 08:59:42.249234 (+ 286us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.249838 (+ 604us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.249995 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.259994 11967 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:42.261154 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2325) took 2176 ms. Trace:
I20250113 08:59:42.261514 11967 rpcz_store.cc:276] 0113 08:59:40.084810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.259179 (+2174369us) service_pool.cc:225] Handling call
0113 08:59:42.259551 (+ 372us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.260385 (+ 834us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.261084 (+ 699us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:42.256942 12008 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.006s sys 0.011s
I20250113 08:59:42.262897 12723 raft_consensus.cc:1235] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 6. Current term is 7. Ops: []
W20250113 08:59:42.263023 12008 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.683s user 0.007s sys 0.012s
W20250113 08:59:42.264075 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 683817us
I20250113 08:59:42.264786 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2305) took 2405 ms. Trace:
I20250113 08:59:42.265311 11965 rpcz_store.cc:276] 0113 08:59:39.859185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.254767 (+2395582us) service_pool.cc:225] Handling call
0113 08:59:42.255064 (+ 297us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.264457 (+ 9393us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.264703 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.267663 11967 raft_consensus.cc:1270] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250113 08:59:42.269559 11967 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2311) took 1822 ms. Trace:
I20250113 08:59:42.269822 11961 raft_consensus.cc:1235] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20250113 08:59:42.269928 11967 rpcz_store.cc:276] 0113 08:59:40.446972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.262188 (+1815216us) service_pool.cc:225] Handling call
0113 08:59:42.267316 (+ 5128us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.269221 (+ 1905us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.269494 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.291394 15187 raft_consensus.cc:3049] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:42.292007 15187 raft_consensus.cc:738] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:42.293742 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:42.293923 15129 leader_election.cc:290] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.294555 12723 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:42.294826 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2002) took 3795 ms. Trace:
I20250113 08:59:42.295192 11887 rpcz_store.cc:276] 0113 08:59:38.499899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.232377 (+1732478us) service_pool.cc:225] Handling call
0113 08:59:41.625377 (+1393000us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.699205 (+ 73828us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.294640 (+595435us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54000d7d50. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:42.294768 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3202710784}
I20250113 08:59:42.295812 11965 raft_consensus.cc:1270] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250113 08:59:42.248543 15214 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:42.310552 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54924 (request call id 549) took 2954 ms. Trace:
I20250113 08:59:42.311164 11961 rpcz_store.cc:276] 0113 08:59:39.356685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.244582 (+2887897us) service_pool.cc:225] Handling call
0113 08:59:42.269466 (+ 24884us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:42.310037 (+ 40571us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.310463 (+ 426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.263484 12722 consensus_queue.cc:260] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.325572 12722 raft_consensus.cc:3054] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Advancing to term 10
I20250113 08:59:42.327647 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2312) took 1573 ms. Trace:
I20250113 08:59:42.328194 11965 rpcz_store.cc:276] 0113 08:59:40.723491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.266126 (+1542635us) service_pool.cc:225] Handling call
0113 08:59:42.266475 (+ 349us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.296161 (+ 29686us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.296346 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:59:42.329772 12685 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 493115us
I20250113 08:59:42.341130 11965 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20250113 08:59:42.342196 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2332) took 1043 ms. Trace:
I20250113 08:59:42.342577 11965 rpcz_store.cc:276] 0113 08:59:41.298504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.335499 (+1036995us) service_pool.cc:225] Handling call
0113 08:59:42.335852 (+ 353us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.341850 (+ 5998us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.342129 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.343940 11961 raft_consensus.cc:1235] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:42.344707 11961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2331) took 1485 ms. Trace:
I20250113 08:59:42.345084 11961 rpcz_store.cc:276] 0113 08:59:40.859305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.312123 (+1452818us) service_pool.cc:225] Handling call
0113 08:59:42.343704 (+ 31581us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.344430 (+ 726us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.344631 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.351388 11959 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250113 08:59:42.351754 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:42.352439 11959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2315) took 5757 ms. Trace:
I20250113 08:59:42.352995 11959 rpcz_store.cc:276] 0113 08:59:36.595479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.118499 (+5523020us) service_pool.cc:225] Handling call
0113 08:59:42.180222 (+ 61723us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.352051 (+171829us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.352309 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.354729 11959 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250113 08:59:42.239353 15268 raft_consensus.cc:513] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.352653 12723 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:42.292996 15187 consensus_queue.cc:260] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:42.403745 15187 raft_consensus.cc:3054] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:42.406069 12717 raft_consensus.cc:2463] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:42.392310 11890 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
W20250113 08:59:42.410202 15328 log.cc:926] Time spent T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 1.962s user 0.011s sys 0.000s
I20250113 08:59:42.410679 11963 raft_consensus.cc:1235] T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:42.411558 11963 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2307) took 7552 ms. Trace:
I20250113 08:59:42.410178 15065 raft_consensus.cc:513] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.411984 11963 rpcz_store.cc:276] 0113 08:59:34.859238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.045075 (+6185837us) service_pool.cc:225] Handling call
0113 08:59:41.072775 (+ 27700us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.410463 (+1337688us) spinlock_profiling.cc:243] Waited 63.8 ms on lock 0x7b5400147e20. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336599990 00007fc33660b300 00007fc33660c371 00007fc336608edf 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc
0113 08:59:42.411213 (+ 750us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.411478 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146762496}
I20250113 08:59:42.393265 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:42.414256 15065 leader_election.cc:290] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.393225 12722 raft_consensus.cc:1270] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
W20250113 08:59:42.401980 15381 log.cc:926] Time spent T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
I20250113 08:59:42.380064 14966 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:42.418305 12722 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250113 08:59:42.337915 15041 raft_consensus.cc:695] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:42.388712 15129 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:42.402639 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1804) took 2227 ms. Trace:
I20250113 08:59:42.424363 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:42.425442 11934 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.011s sys 0.013s
W20250113 08:59:42.425792 11934 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.586s user 0.011s sys 0.013s
I20250113 08:59:42.416599 11890 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 8.
I20250113 08:59:42.388464 14994 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.418243 14966 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:42.426059 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 585921us
I20250113 08:59:42.422129 12036 rpcz_store.cc:276] 0113 08:59:40.175497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.372425 (+196928us) service_pool.cc:225] Handling call
0113 08:59:40.392370 (+ 19945us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:40.501623 (+109253us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:40.501671 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:40.508289 (+ 6618us) log.cc:844] Serialized 52 byte log entry
0113 08:59:40.585109 (+ 76820us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:40.594459 (+ 9350us) log.cc:844] Serialized 14 byte log entry
0113 08:59:40.600684 (+ 6225us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:40.600812 (+ 128us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:40.787530 (+186718us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b180043d8c0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc3336d1204 00007fc3336d45b1 00007fc3336d4569 00007fc3336d44f9 00007fc3336d44c1 00007fc3336d37bd 00007fc33c70d234 00007fc33c70d069 00007fc3336b38d6
0113 08:59:42.336712 (+1549182us) spinlock_profiling.cc:243] Waited 39.5 ms on lock 0x7b24001e6248. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc333528264 00007fc33352636b 00007fc3335b4b19 00007fc3335b47a5 00007fc3335b3d00 00007fc3362b3f88 00007fc3362b46e8 00007fc33628bf98 00007fc3362873aa 00007fc3362867f6
0113 08:59:42.402365 (+ 65653us) raft_consensus.cc:1680] finished
0113 08:59:42.402393 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.402572 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89064,"prepare.queue_time_us":24761,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":3683,"raft.queue_time_us":99843,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":12745,"spinlock_wait_cycles":98173952,"thread_start_us":55250,"threads_started":1,"wal-append.queue_time_us":127354}
I20250113 08:59:42.432893 11934 leader_election.cc:304] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:42.421371 15302 consensus_queue.cc:1046] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.005s
I20250113 08:59:42.434203 15214 raft_consensus.cc:513] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.438624 12723 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
W20250113 08:59:42.432003 15150 consensus_meta.cc:287] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.727s user 0.014s sys 0.000s
I20250113 08:59:42.441012 12722 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250113 08:59:42.447690 11891 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.008s sys 0.019s
W20250113 08:59:42.448262 11891 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.008s sys 0.019s
I20250113 08:59:42.451756 12038 raft_consensus.cc:1270] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20250113 08:59:42.453913 12038 pending_rounds.cc:77] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 2
I20250113 08:59:42.454236 12038 pending_rounds.cc:99] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20250113 08:59:42.454528 12038 raft_consensus.cc:2883] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:42.458863 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 566814us
I20250113 08:59:42.459672 12722 raft_consensus.cc:1270] T 69aa379773c047589ae2362850997a47 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250113 08:59:42.459724 12686 leader_election.cc:400] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8.
I20250113 08:59:42.463344 12686 leader_election.cc:403] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20250113 08:59:42.464254 12686 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:42.435046 15129 raft_consensus.cc:513] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.452483 15065 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:42.449018 14879 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.028s
I20250113 08:59:42.467010 15049 raft_consensus.cc:2798] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:42.466928 14981 raft_consensus.cc:3054] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 8
I20250113 08:59:42.461102 14966 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.482808 14994 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.472102 12686 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:42.490927 11858 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.012s sys 0.015s
I20250113 08:59:42.492616 12686 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:42.458660 15150 consensus_queue.cc:1046] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20250113 08:59:42.497179 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1927) took 2047 ms. Trace:
I20250113 08:59:42.498101 11891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1932) took 1743 ms. Trace:
W20250113 08:59:42.493214 11858 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.012s sys 0.015s
I20250113 08:59:42.497606 11884 rpcz_store.cc:276] 0113 08:59:40.450658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.061096 (+610438us) service_pool.cc:225] Handling call
0113 08:59:41.077626 (+ 16530us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:41.086147 (+ 8521us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:41.505922 (+419775us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.003767 (+497845us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:42.032782 (+ 29015us) log.cc:844] Serialized 26 byte log entry
0113 08:59:42.032967 (+ 185us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:42.033309 (+ 342us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.043550 (+ 10241us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.043651 (+ 101us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:42.496969 (+453318us) raft_consensus.cc:1680] finished
0113 08:59:42.496991 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.497118 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11152,"prepare.queue_time_us":536808,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":6626,"raft.queue_time_us":283541,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":26720,"thread_start_us":789756,"threads_started":2,"wal-append.queue_time_us":697290}
W20250113 08:59:42.492681 15373 log.cc:926] Time spent T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.760s user 0.003s sys 0.000s
I20250113 08:59:42.486265 15065 raft_consensus.cc:513] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:42.489161 15268 leader_election.cc:290] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.494480 15049 raft_consensus.cc:695] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:42.528565 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc84bb347ec2413e9188b7253b0812ab" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.489527 15302 consensus_queue.cc:1035] T 663c56b937924e7abebd7025ca892c1e P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
W20250113 08:59:42.533370 11888 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.003s sys 0.055s
W20250113 08:59:42.548729 11888 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.003s sys 0.055s
I20250113 08:59:42.548821 15391 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:42.549527 15391 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:42.550787 11858 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:42.551612 15199 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
W20250113 08:59:42.518774 12684 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.017s sys 0.004s
I20250113 08:59:42.474658 15041 consensus_queue.cc:237] T a23cb6607d3a463293b397049ef2d690 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:42.534302 11859 leader_election.cc:336] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.839s (SENT)
I20250113 08:59:42.506114 15247 raft_consensus.cc:491] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:42.569762 12684 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.666s user 0.019s sys 0.006s
I20250113 08:59:42.570058 11859 leader_election.cc:304] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
W20250113 08:59:42.570410 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 707681us
I20250113 08:59:42.579408 15065 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:42.580308 15272 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.007s sys 0.019s
W20250113 08:59:42.580660 15272 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.007s sys 0.020s
W20250113 08:59:42.582113 15384 log.cc:926] Time spent T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
W20250113 08:59:42.582264 15385 log.cc:926] Time spent T cee55da481c64a6d885aa3ab442761cd P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250113 08:59:42.587123 15049 consensus_queue.cc:237] T a2bbae0b61a948339e4c443b9237ad87 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.589387 14879 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250113 08:59:42.591248 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1930) took 1885 ms. Trace:
I20250113 08:59:42.574380 15268 consensus_queue.cc:1046] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20250113 08:59:42.591665 11893 rpcz_store.cc:276] 0113 08:59:40.705763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.810393 (+1104630us) service_pool.cc:225] Handling call
0113 08:59:41.810760 (+ 367us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:41.811104 (+ 344us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:41.811138 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:41.816283 (+ 5145us) log.cc:844] Serialized 52 byte log entry
0113 08:59:42.269768 (+453485us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:42.269810 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.269881 (+ 71us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:42.590999 (+321118us) raft_consensus.cc:1680] finished
0113 08:59:42.591024 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.591170 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":584,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":5540,"thread_start_us":431801,"threads_started":1,"wal-append.queue_time_us":659426}
I20250113 08:59:42.576247 15150 raft_consensus.cc:3049] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Stepping down as leader of term 5
W20250113 08:59:42.593171 15358 log.cc:926] Time spent T 8b1c553832b94166a0a6089104a74f6c P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.971s user 0.008s sys 0.000s
I20250113 08:59:42.593300 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.594054 11884 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 7.
W20250113 08:59:42.597800 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 63551.4us
I20250113 08:59:42.602135 12684 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
W20250113 08:59:42.606496 15190 consensus_meta.cc:287] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.534s user 0.008s sys 0.003s
I20250113 08:59:42.607007 15190 raft_consensus.cc:513] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.551592 15391 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.606124 15314 consensus_queue.cc:1046] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:42.552212 15199 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.614037 12684 leader_election.cc:304] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:42.614197 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:42.617100 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.558951 15053 raft_consensus.cc:491] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:42.554049 15214 leader_election.cc:290] T 3e1a8409abe745918b8f11dc8590376a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 9 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:42.577533 15390 raft_consensus.cc:2743] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:42.593382 15150 raft_consensus.cc:738] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:42.604640 15187 raft_consensus.cc:2743] T 07a14a3957e44925ae0925b645d220eb P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:42.638722 12684 leader_election.cc:304] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:42.644776 12684 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:42.660362 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 719) took 1355 ms. Trace:
I20250113 08:59:42.647495 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.618918 15190 leader_election.cc:290] T 099cf849e7414101a8741339c72fa73e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:42.576846 15247 raft_consensus.cc:513] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.664435 15302 raft_consensus.cc:491] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:42.648234 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.613205 15070 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
W20250113 08:59:42.667446 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 275868us
I20250113 08:59:42.674753 12684 leader_election.cc:304] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:42.616676 15065 consensus_queue.cc:1035] T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250113 08:59:42.698720 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2027) took 2686 ms. Trace:
I20250113 08:59:42.628913 15391 raft_consensus.cc:3049] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:42.620034 15268 consensus_queue.cc:1035] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.049s
I20250113 08:59:42.613209 15261 raft_consensus.cc:2743] T 7dda9aa8769e4146a4dabeadd588c734 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:42.720748 11960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2308) took 7837 ms. Trace:
I20250113 08:59:42.721176 11960 rpcz_store.cc:276] 0113 08:59:34.883153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.046825 (+6163672us) service_pool.cc:225] Handling call
0113 08:59:41.053945 (+ 7120us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.064317 (+ 10372us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.720455 (+1656138us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b5400148820. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336599990 00007fc33660cb42 00007fc336608f26 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782
0113 08:59:42.720655 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52107008}
I20250113 08:59:42.689963 15054 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250113 08:59:42.703332 15256 consensus_meta.cc:287] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff: Time spent flushing consensus metadata: real 0.507s user 0.012s sys 0.000s
I20250113 08:59:42.666276 15393 raft_consensus.cc:2743] T a43de3c8f06b48358bcee58f9567bb3b P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:42.715548 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.647104 15214 raft_consensus.cc:2798] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 12 FOLLOWER]: Leader election won for term 12
I20250113 08:59:42.690686 12719 rpcz_store.cc:276] 0113 08:59:41.304797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.191878 (+887081us) service_pool.cc:225] Handling call
0113 08:59:42.220074 (+ 28196us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.220300 (+ 226us) raft_consensus.cc:1529] Early marking committed up to index 8
0113 08:59:42.238999 (+ 18699us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.639572 (+400573us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:42.639628 (+ 56us) raft_consensus.cc:1640] Marking committed up to 8
0113 08:59:42.645766 (+ 6138us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.645859 (+ 93us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.660267 (+ 14408us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7592,"prepare.queue_time_us":19073,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":36,"raft.queue_time_us":7767,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":26070,"thread_start_us":392433,"threads_started":1}
I20250113 08:59:42.701479 11892 rpcz_store.cc:276] 0113 08:59:40.011858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:40.853968 (+842110us) service_pool.cc:225] Handling call
0113 08:59:40.855389 (+ 1421us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:40.857450 (+ 2061us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:40.857481 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:40.857929 (+ 448us) log.cc:844] Serialized 26 byte log entry
0113 08:59:41.106938 (+249009us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:41.538295 (+431357us) log.cc:844] Serialized 14 byte log entry
0113 08:59:41.540406 (+ 2111us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:41.540514 (+ 108us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:42.592621 (+1052107us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:42.680994 (+ 88373us) raft_consensus.cc:1680] finished
0113 08:59:42.698463 (+ 17469us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.698643 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1692,"prepare.queue_time_us":516108,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":1909,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":11704,"spinlock_wait_cycles":30555520,"thread_start_us":673810,"threads_started":2,"wal-append.queue_time_us":764249}
I20250113 08:59:42.727159 12008 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:42.667727 15251 raft_consensus.cc:2743] T 67a8b82ef293439e815f49903f8e5dbb P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:42.638461 15053 raft_consensus.cc:513] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.719738 15268 consensus_queue.cc:1225] T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 37bd4bed3ee6423c8dac0cef3bf6882a log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250113 08:59:42.738632 12008 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:42.730103 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:42.736788 11884 raft_consensus.cc:2388] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:42.743575 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.744344 11892 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:42.748227 11893 raft_consensus.cc:2388] T 099cf849e7414101a8741339c72fa73e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:42.752488 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:42.753005 15199 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.753547 11884 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:42.756572 15146 raft_consensus.cc:3054] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:42.758661 15247 leader_election.cc:290] T de29a6701ddb4906a7eff4d34bf97ab5 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.732049 15256 raft_consensus.cc:513] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.759533 14783 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:42.761266 15256 leader_election.cc:290] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:42.723313 15150 consensus_queue.cc:260] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.763368 15150 raft_consensus.cc:3054] T b6a40dc5e61345558c943df4be928468 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:42.764761 15303 raft_consensus.cc:3049] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:42.700601 15302 raft_consensus.cc:513] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:42.733891 15189 raft_consensus.cc:2798] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:42.778287 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:42.715014 15391 raft_consensus.cc:738] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
I20250113 08:59:42.782684 11893 raft_consensus.cc:1270] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250113 08:59:42.783206 12719 raft_consensus.cc:1270] T 705a99b7862446be849d4c843d8084ee P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250113 08:59:42.776578 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 325332us
I20250113 08:59:42.770751 15303 raft_consensus.cc:738] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:42.783349 11893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1939) took 1237 ms. Trace:
I20250113 08:59:42.785574 11893 rpcz_store.cc:276] 0113 08:59:41.546468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.749041 (+1202573us) service_pool.cc:225] Handling call
0113 08:59:42.782352 (+ 33311us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.783065 (+ 713us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.783292 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.783776 12719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 848) took 1087 ms. Trace:
I20250113 08:59:42.788214 12719 rpcz_store.cc:276] 0113 08:59:41.695891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.740411 (+1044520us) service_pool.cc:225] Handling call
0113 08:59:42.750135 (+ 9724us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.783550 (+ 33415us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.783728 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.779157 11892 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:42.791322 11892 raft_consensus.cc:1270] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250113 08:59:42.791821 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1940) took 1077 ms. Trace:
I20250113 08:59:42.793118 11892 rpcz_store.cc:276] 0113 08:59:41.714370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.790654 (+1076284us) service_pool.cc:225] Handling call
0113 08:59:42.790987 (+ 333us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.791627 (+ 640us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.791774 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.795459 11892 raft_consensus.cc:1235] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:42.796419 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2030) took 1073 ms. Trace:
I20250113 08:59:42.797039 11892 rpcz_store.cc:276] 0113 08:59:41.723002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.794690 (+1071688us) service_pool.cc:225] Handling call
0113 08:59:42.795177 (+ 487us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.796095 (+ 918us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.796347 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.799587 11892 raft_consensus.cc:1270] T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250113 08:59:42.800720 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2035) took 1076 ms. Trace:
I20250113 08:59:42.801299 11892 rpcz_store.cc:276] 0113 08:59:41.724299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.798393 (+1074094us) service_pool.cc:225] Handling call
0113 08:59:42.798985 (+ 592us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.800300 (+ 1315us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.800640 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.803619 11892 raft_consensus.cc:1270] T 7d7522e543294e28ac66b2faf6001cbe P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250113 08:59:42.804512 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2036) took 1068 ms. Trace:
I20250113 08:59:42.805163 11892 rpcz_store.cc:276] 0113 08:59:41.745213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.802571 (+1057358us) service_pool.cc:225] Handling call
0113 08:59:42.803066 (+ 495us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.804179 (+ 1113us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.804408 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:59:42.815930 12721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 919) took 1306 ms. Trace:
I20250113 08:59:42.819072 12721 rpcz_store.cc:276] 0113 08:59:41.509917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.212634 (+702717us) service_pool.cc:225] Handling call
0113 08:59:42.232152 (+ 19518us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.232485 (+ 333us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:42.232521 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:42.232898 (+ 377us) log.cc:844] Serialized 26 byte log entry
0113 08:59:42.622383 (+389485us) raft_consensus.cc:1640] Marking committed up to 9
0113 08:59:42.665148 (+ 42765us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.665511 (+ 363us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.665597 (+ 86us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:42.814977 (+149380us) raft_consensus.cc:1680] finished
0113 08:59:42.815718 (+ 741us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.815861 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9244,"prepare.queue_time_us":35887,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":12853,"raft.queue_time_us":938,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":14846,"spinlock_wait_cycles":14034304,"thread_start_us":389146,"threads_started":1,"wal-append.queue_time_us":501387}
I20250113 08:59:42.860988 11886 raft_consensus.cc:2463] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:42.861934 15104 raft_consensus.cc:491] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:42.862583 15104 raft_consensus.cc:513] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
W20250113 08:59:42.875645 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 137097us
I20250113 08:59:42.877085 12008 leader_election.cc:304] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:42.872841 15053 leader_election.cc:290] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.871011 15418 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:42.910946 12006 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:42.875659 15416 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:42.918373 15416 raft_consensus.cc:491] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:42.968628 15416 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:42.968914 15302 leader_election.cc:290] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:42.971530 15053 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:42.972153 15053 raft_consensus.cc:513] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:42.974109 11892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2039) took 1117 ms. Trace:
I20250113 08:59:42.975288 12037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1821) took 2190 ms. Trace:
I20250113 08:59:42.977715 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2368) took 1953 ms. Trace:
I20250113 08:59:42.978353 12039 rpcz_store.cc:276] 0113 08:59:41.024458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.067778 (+ 43320us) service_pool.cc:225] Handling call
0113 08:59:41.083290 (+ 15512us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.083558 (+ 268us) raft_consensus.cc:1529] Early marking committed up to index 4
0113 08:59:41.385570 (+302012us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.826217 (+1440647us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.977175 (+150958us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:42.977233 (+ 58us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:42.977303 (+ 70us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.977441 (+ 138us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.977661 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1456751,"prepare.queue_time_us":1492000,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":1754,"raft.queue_time_us":254,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":7198,"thread_start_us":423668,"threads_started":3}
I20250113 08:59:42.980893 12726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 912) took 4577 ms. Trace:
I20250113 08:59:42.982652 11968 raft_consensus.cc:1235] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 5. Current term is 7. Ops: [5.4-5.4]
I20250113 08:59:42.976013 12037 rpcz_store.cc:276] 0113 08:59:40.784887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.019698 (+234811us) service_pool.cc:225] Handling call
0113 08:59:41.019985 (+ 287us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.020175 (+ 190us) raft_consensus.cc:1529] Early marking committed up to index 4
0113 08:59:41.223839 (+203664us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.808854 (+1585015us) spinlock_profiling.cc:243] Waited 3.03 ms on lock 0x7b1800408ca0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc33364893a 00007fc33cb04be9 00007fc333509ad0 00007fc3336ce304 00007fc3336cdbf5 00007fc339fde107 00007fc3366138f7 00007fc33662300f 00007fc336622f88
0113 08:59:42.809070 (+ 216us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.974956 (+165886us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:42.975010 (+ 54us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:42.975053 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.975121 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.975231 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106907,"prepare.queue_time_us":1646277,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":2517,"raft.queue_time_us":75,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":12554,"spinlock_wait_cycles":6974336,"thread_start_us":334964,"threads_started":3}
I20250113 08:59:42.982896 12726 rpcz_store.cc:276] 0113 08:59:38.403473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.414450 (+ 10977us) service_pool.cc:225] Handling call
0113 08:59:41.431649 (+3017199us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.549966 (+118317us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.980049 (+1430083us) spinlock_profiling.cc:243] Waited 3 s on lock 0x7b54006211d0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:42.980585 (+ 536us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6907380096}
I20250113 08:59:42.850037 15391 consensus_queue.cc:260] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:42.995625 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 118652us
I20250113 08:59:42.996908 12008 leader_election.cc:304] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:42.976990 15424 raft_consensus.cc:491] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:42.976184 14879 raft_consensus.cc:491] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:42.998554 11884 raft_consensus.cc:2463] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 8.
W20250113 08:59:42.992787 15406 log.cc:926] Time spent T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20250113 08:59:42.987768 14981 raft_consensus.cc:2743] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863) with higher term. Message: Invalid argument: T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8.
I20250113 08:59:43.005414 11884 pending_rounds.cc:77] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Aborting all ops after (but not including) 3
W20250113 08:59:43.006883 15415 log.cc:926] Time spent T 8133c246c9d74113aec7a002df391ef7 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
W20250113 08:59:43.008162 15412 log.cc:926] Time spent T a47a39083de44711b040bcf422953893 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20250113 08:59:43.009250 15053 leader_election.cc:290] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:43.009933 15391 raft_consensus.cc:3054] T faa96abb4c9940e4aa27327deeb68af0 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:43.015578 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1937) took 1923 ms. Trace:
I20250113 08:59:43.016158 11889 rpcz_store.cc:276] 0113 08:59:41.091733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.127828 (+1036095us) service_pool.cc:225] Handling call
0113 08:59:42.128303 (+ 475us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.492782 (+364479us) raft_consensus.cc:1529] Early marking committed up to index 2
0113 08:59:42.510566 (+ 17784us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.781936 (+271370us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:42.787724 (+ 5788us) log.cc:844] Serialized 26 byte log entry
0113 08:59:42.787878 (+ 154us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:42.787908 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.787970 (+ 62us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:43.015280 (+227310us) raft_consensus.cc:1680] finished
0113 08:59:43.015306 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.015490 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2563,"prepare.queue_time_us":17623,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":98648,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"thread_start_us":554616,"threads_started":2,"wal-append.queue_time_us":357833}
I20250113 08:59:43.005956 11884 pending_rounds.cc:99] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250113 08:59:42.908121 15303 consensus_queue.cc:260] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.027346 11884 raft_consensus.cc:2883] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:43.030418 11884 pending_rounds.cc:99] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20250113 08:59:43.031652 11884 raft_consensus.cc:2883] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:43.005447 15424 raft_consensus.cc:513] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.004529 15418 raft_consensus.cc:491] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:43.004390 14879 raft_consensus.cc:513] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:43.060106 11859 leader_election.cc:336] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 2.305s (SENT)
W20250113 08:59:43.062952 12724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 842) took 1367 ms (client timeout 1666 ms). Trace:
W20250113 08:59:43.065055 12724 rpcz_store.cc:269] 0113 08:59:41.690776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.722088 (+ 31312us) service_pool.cc:225] Handling call
0113 08:59:43.058201 (+1336113us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b540068ddd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.058318 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112586624}
I20250113 08:59:43.068001 12720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38504 (request call id 923) took 1521 ms. Trace:
W20250113 08:59:43.055485 15410 log.cc:926] Time spent T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
W20250113 08:59:43.060935 15409 log.cc:926] Time spent T 637bd1e4126142b481c15befc679b5ac P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.192s user 0.001s sys 0.000s
I20250113 08:59:43.068655 12720 rpcz_store.cc:276] 0113 08:59:41.546795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.220941 (+674146us) service_pool.cc:225] Handling call
0113 08:59:42.229624 (+ 8683us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.230044 (+ 420us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:42.230076 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:42.230661 (+ 585us) log.cc:844] Serialized 26 byte log entry
0113 08:59:42.731144 (+500483us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:42.731206 (+ 62us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.732171 (+ 965us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:43.067673 (+335502us) raft_consensus.cc:1680] finished
0113 08:59:43.067711 (+ 38us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.067915 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1165,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":500042,"threads_started":1,"wal-append.queue_time_us":592891}
I20250113 08:59:43.050683 15053 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.083293 15053 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.082182 15433 raft_consensus.cc:2743] T 8133c246c9d74113aec7a002df391ef7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:43.082432 12034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33876 (request call id 2370) took 1928 ms. Trace:
I20250113 08:59:43.085718 12034 rpcz_store.cc:276] 0113 08:59:41.154098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.154239 (+ 141us) service_pool.cc:225] Handling call
0113 08:59:41.235821 (+ 81582us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:41.246408 (+ 10587us) raft_consensus.cc:1529] Early marking committed up to index 10
0113 08:59:42.817274 (+1570866us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.990729 (+173455us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:43.004034 (+ 13305us) log.cc:844] Serialized 26 byte log entry
0113 08:59:43.004237 (+ 203us) raft_consensus.cc:1640] Marking committed up to 10
0113 08:59:43.004285 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.004411 (+ 126us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:43.082136 (+ 77725us) raft_consensus.cc:1680] finished
0113 08:59:43.082162 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.082349 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1642517,"prepare.queue_time_us":1618453,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":104060,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":5118,"thread_start_us":112442,"threads_started":1,"wal-append.queue_time_us":208315}
I20250113 08:59:43.054885 15434 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250113 08:59:43.088616 12006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 177769us
I20250113 08:59:43.032557 15413 consensus_queue.cc:1046] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20250113 08:59:43.088915 15435 raft_consensus.cc:2743] T 531b33203b294592b3a1275dc74ab4f3 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:43.095227 15424 leader_election.cc:290] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:43.097414 15416 raft_consensus.cc:513] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.069523 15422 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.100548 15416 leader_election.cc:290] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.100462 15422 raft_consensus.cc:513] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
W20250113 08:59:43.102079 15379 log.cc:926] Time spent T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.870s user 0.001s sys 0.002s
I20250113 08:59:43.102748 15422 leader_election.cc:290] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:43.104189 15303 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:43.110088 11887 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1938) took 1815 ms. Trace:
I20250113 08:59:43.110505 11887 rpcz_store.cc:276] 0113 08:59:41.294260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.296027 (+1001767us) service_pool.cc:225] Handling call
0113 08:59:42.336358 (+ 40331us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.339690 (+ 3332us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:42.339721 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:42.378391 (+ 38670us) log.cc:844] Serialized 26 byte log entry
0113 08:59:42.663439 (+285048us) raft_consensus.cc:1640] Marking committed up to 3
0113 08:59:42.663513 (+ 74us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.663622 (+ 109us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:43.109846 (+446224us) raft_consensus.cc:1680] finished
0113 08:59:43.109874 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.110029 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2914,"raft.queue_time_us":12055,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":19335,"thread_start_us":241088,"threads_started":1,"wal-append.queue_time_us":351746}
W20250113 08:59:43.111647 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 114821us
I20250113 08:59:43.112387 12008 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:43.113857 12008 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:43.089493 12006 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.059069 15418 raft_consensus.cc:513] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.092818 15053 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:43.115092 11893 raft_consensus.cc:1235] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:43.118183 11887 raft_consensus.cc:1270] T 272c58aa3da64d86aa8a3e3890e3a249 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Refusing update from remote peer 37bd4bed3ee6423c8dac0cef3bf6882a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20250113 08:59:43.119946 11887 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [term 10 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 9. Current term is 10. Ops: []
I20250113 08:59:43.088744 15432 raft_consensus.cc:491] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:43.121762 12040 raft_consensus.cc:1235] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 10 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 9. Current term is 10. Ops: []
I20250113 08:59:43.123116 11887 raft_consensus.cc:1235] T 218ea9985e1d40a48892a8e01aaaef39 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:43.126502 15437 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:43.122267 15432 raft_consensus.cc:513] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.117074 15422 raft_consensus.cc:3054] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:43.127005 15437 raft_consensus.cc:491] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:43.127537 15437 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:43.127972 12008 leader_election.cc:304] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:43.090282 15434 raft_consensus.cc:491] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:43.090288 15302 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.134104 15442 consensus_queue.cc:1035] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20250113 08:59:43.137897 15432 leader_election.cc:290] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:43.132357 15302 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.140002 15302 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:43.129902 15434 raft_consensus.cc:3054] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:43.141391 15438 raft_consensus.cc:2798] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:43.142007 15438 raft_consensus.cc:695] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:43.142953 15438 consensus_queue.cc:237] T de29a6701ddb4906a7eff4d34bf97ab5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.145364 15437 raft_consensus.cc:513] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.147102 15432 consensus_queue.cc:1035] T 992519c5663649019a1ff9753075ddc7 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250113 08:59:43.148028 15437 leader_election.cc:290] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.148480 15432 consensus_queue.cc:1035] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250113 08:59:43.150552 15341 leader_election.cc:290] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:43.152305 11887 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:43.159690 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.160408 12724 raft_consensus.cc:2463] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 8.
I20250113 08:59:43.161492 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:43.161597 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:43.162062 12724 raft_consensus.cc:3054] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:43.162453 12722 raft_consensus.cc:1270] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:43.163774 15224 raft_consensus.cc:3049] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:43.164194 15224 raft_consensus.cc:738] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:43.164351 15440 consensus_queue.cc:1035] T 705a99b7862446be849d4c843d8084ee P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20250113 08:59:43.165103 15224 consensus_queue.cc:260] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.160079 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:43.166923 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 492277us
I20250113 08:59:43.162082 12725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38492 (request call id 717) took 1857 ms. Trace:
I20250113 08:59:43.170394 12725 rpcz_store.cc:276] 0113 08:59:41.304197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.494114 (+189917us) service_pool.cc:225] Handling call
0113 08:59:43.161892 (+1667778us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54005ae454. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.162029 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":486357248}
W20250113 08:59:43.169118 15007 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s user 0.004s sys 0.024s
W20250113 08:59:43.173456 15007 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s user 0.004s sys 0.024s
I20250113 08:59:43.174132 11965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:54914 (request call id 11) took 3724 ms. Trace:
I20250113 08:59:43.174247 12719 raft_consensus.cc:1235] T b6a40dc5e61345558c943df4be928468 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 LEADER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:43.174517 11965 rpcz_store.cc:276] 0113 08:59:39.453383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.357700 (+2904317us) service_pool.cc:225] Handling call
0113 08:59:43.173857 (+816157us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400380254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660f7c5 00007fc33b568f29 00007fc334c3a714 00007fc334c3a6a2 00007fc334c3a5d1 00007fc334c3a54c 00007fc334c39802 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.174034 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1438521088}
I20250113 08:59:43.176751 11890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 2037) took 1321 ms. Trace:
I20250113 08:59:43.177140 12726 raft_consensus.cc:1235] T 80cc6c97986444348b1837c27c41338a P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: []
I20250113 08:59:43.178440 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1835) took 1366 ms. Trace:
I20250113 08:59:43.178833 12038 rpcz_store.cc:276] 0113 08:59:41.812051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.077151 (+265100us) service_pool.cc:225] Handling call
0113 08:59:42.077932 (+ 781us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:42.460046 (+382114us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.178187 (+718141us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54001fccd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:43.178384 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":851997952}
I20250113 08:59:42.499179 11891 rpcz_store.cc:276] 0113 08:59:40.754604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.835487 (+1080883us) service_pool.cc:225] Handling call
0113 08:59:41.849065 (+ 13578us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.849341 (+ 276us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:41.870462 (+ 21121us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.497713 (+627251us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:42.497775 (+ 62us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:42.497824 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.497898 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.498011 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49320,"prepare.queue_time_us":167967,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":48954,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":14763,"thread_start_us":577951,"threads_started":1}
I20250113 08:59:43.154762 15302 consensus_queue.cc:1035] T 69aa379773c047589ae2362850997a47 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.031s
W20250113 08:59:43.186435 15380 log.cc:926] Time spent T 87a2e0ccdce34284b13fa0924326b262 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.957s user 0.002s sys 0.000s
I20250113 08:59:43.189657 14879 leader_election.cc:290] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:43.190551 11888 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1929) took 2491 ms. Trace:
I20250113 08:59:43.191591 11884 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52600 (request call id 1942) took 1331 ms. Trace:
I20250113 08:59:43.192117 11884 rpcz_store.cc:276] 0113 08:59:41.860002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:43.004664 (+1144662us) service_pool.cc:225] Handling call
0113 08:59:43.005084 (+ 420us) raft_consensus.cc:1466] Updating replica for 3 ops
0113 08:59:43.039248 (+ 34164us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:43.039292 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0113 08:59:43.040240 (+ 948us) log.cc:844] Serialized 78 byte log entry
0113 08:59:43.151538 (+111298us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:43.151577 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.151644 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:43.191283 (+ 39639us) raft_consensus.cc:1680] finished
0113 08:59:43.191313 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.191517 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":145,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":13463,"thread_start_us":87703,"threads_started":1,"wal-append.queue_time_us":136575}
I20250113 08:59:43.195614 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.177073 11890 rpcz_store.cc:276] 0113 08:59:41.855877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.050499 (+194622us) service_pool.cc:225] Handling call
0113 08:59:43.176577 (+1126078us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b5400060ed4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.176691 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":703735040}
I20250113 08:59:43.197427 12724 raft_consensus.cc:2463] T badac2289c8b4a10915cca20182fc423 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
I20250113 08:59:43.172871 12008 leader_election.cc:304] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a
I20250113 08:59:43.197803 12006 leader_election.cc:304] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:43.190944 11888 rpcz_store.cc:276] 0113 08:59:40.699152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.429595 (+730443us) service_pool.cc:225] Handling call
0113 08:59:41.429951 (+ 356us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:41.549934 (+119983us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:41.549983 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:41.550497 (+ 514us) log.cc:844] Serialized 26 byte log entry
0113 08:59:41.864699 (+314202us) raft_consensus.cc:1640] Marking committed up to 7
0113 08:59:42.006881 (+142182us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.549404 (+542523us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.549509 (+ 105us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:43.185757 (+636248us) spinlock_profiling.cc:243] Waited 8.43 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:43.190260 (+ 4503us) raft_consensus.cc:1680] finished
0113 08:59:43.190361 (+ 101us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.190491 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14002,"prepare.queue_time_us":155811,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":158970,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":6164,"spinlock_wait_cycles":19391744,"thread_start_us":815766,"threads_started":2,"wal-append.queue_time_us":585331}
I20250113 08:59:43.198186 15224 raft_consensus.cc:3054] T 7dda9aa8769e4146a4dabeadd588c734 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:43.201743 14879 raft_consensus.cc:2798] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250113 08:59:43.215592 11886 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52586 (request call id 2005) took 4714 ms (client timeout 1643 ms). Trace:
W20250113 08:59:43.216830 11886 rpcz_store.cc:269] 0113 08:59:38.500808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:38.558632 (+ 57824us) service_pool.cc:225] Handling call
0113 08:59:43.215398 (+4656766us) spinlock_profiling.cc:243] Waited 4.25 s on lock 0x7b540037fd54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.215525 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9777477504}
I20250113 08:59:43.218449 11885 raft_consensus.cc:1270] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5e8883f5f83a47f4b9303dd1da4f59c0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:43.218448 12718 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38492 (request call id 714) took 1915 ms. Trace:
I20250113 08:59:43.201890 15302 raft_consensus.cc:2798] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:59:43.219172 12718 rpcz_store.cc:276] 0113 08:59:41.303306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.508643 (+205337us) service_pool.cc:225] Handling call
0113 08:59:41.538146 (+ 29503us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.751819 (+213673us) raft_consensus.cc:1529] Early marking committed up to index 6
0113 08:59:41.751867 (+ 48us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:41.751943 (+ 76us) raft_consensus.cc:1640] Marking committed up to 6
0113 08:59:41.751985 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.218264 (+1466279us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54006e1754. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:43.218302 (+ 38us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.218399 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":490723712}
W20250113 08:59:43.226210 11856 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.341s user 0.013s sys 0.008s
I20250113 08:59:43.227241 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:43.226935 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
W20250113 08:59:43.228376 11856 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.343s user 0.013s sys 0.008s
I20250113 08:59:43.227452 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:43.228861 11890 raft_consensus.cc:3054] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:43.226825 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
W20250113 08:59:43.228891 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 1.34403e+06us
I20250113 08:59:43.227046 11888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:43.231130 11891 raft_consensus.cc:2463] T 79302a9220a149cb81cb2573664290a1 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:43.227732 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:43.232513 11889 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:43.232550 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.227344 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:43.233853 11884 raft_consensus.cc:2388] T 18dede64eda24dff8bfe2f979a4115d5 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 6ea00811b7d44b49b872ac5c03b91487 in this term.
I20250113 08:59:43.236641 15434 raft_consensus.cc:513] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.237898 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.238258 12039 raft_consensus.cc:1235] T 7d7522e543294e28ac66b2faf6001cbe P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:43.240778 12685 leader_election.cc:304] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:43.241675 15372 log.cc:926] Time spent T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 1.471s user 0.003s sys 0.002s
I20250113 08:59:43.242882 15019 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.245555 15061 raft_consensus.cc:491] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.245975 15031 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.247740 12038 raft_consensus.cc:2405] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250113 08:59:43.248919 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.211836 15214 raft_consensus.cc:695] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [term 12 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:43.251267 11886 raft_consensus.cc:1235] T 209260c370164f719208bd1404eef071 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:43.252183 12685 leader_election.cc:304] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.252097 12006 leader_election.cc:304] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.254415 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.255791 11890 raft_consensus.cc:2463] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 7.
W20250113 08:59:43.280794 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 53025.8us
I20250113 08:59:43.282296 12006 leader_election.cc:304] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.283185 12006 leader_election.cc:304] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:43.219409 14879 raft_consensus.cc:491] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:43.284050 11888 raft_consensus.cc:1235] T 6a1ace8d6d3d488583610962dabdee94 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 6. Current term is 7. Ops: []
I20250113 08:59:43.285197 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.291208 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.292246 12033 raft_consensus.cc:2388] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 9: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:43.291316 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.295128 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:43.293028 11959 raft_consensus.cc:2463] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 6.
I20250113 08:59:43.295346 12006 leader_election.cc:304] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:43.296065 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.252151 15302 raft_consensus.cc:695] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:43.298108 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531b33203b294592b3a1275dc74ab4f3" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.297278 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a14a3957e44925ae0925b645d220eb" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.300493 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.304456 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.306596 12036 raft_consensus.cc:2405] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20250113 08:59:43.254357 15393 raft_consensus.cc:2743] T 8f63a7b00ea546f2865a4b5d730e35de P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:43.307757 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:43.312561 12720 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:43.315577 12039 raft_consensus.cc:2463] T 4a9d15ff319f43f0a1dd5ae717393675 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:43.304273 15146 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20250113 08:59:43.309218 15214 consensus_queue.cc:237] T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.291934 15007 consensus_queue.cc:1046] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20250113 08:59:43.312167 15150 raft_consensus.cc:2798] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:43.326947 15007 raft_consensus.cc:3049] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:59:43.327239 15150 raft_consensus.cc:695] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 LEADER]: Becoming Leader. State: Replica: 5e8883f5f83a47f4b9303dd1da4f59c0, State: Running, Role: LEADER
I20250113 08:59:43.304378 15401 consensus_queue.cc:1035] T 54973520c9884989a12742d4eb45994f P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.025s
I20250113 08:59:43.329654 15302 consensus_queue.cc:237] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.331373 15104 leader_election.cc:290] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.332160 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37184 (request call id 644) took 1929 ms. Trace:
I20250113 08:59:43.254868 15391 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.291275 15303 raft_consensus.cc:3054] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:43.292090 15053 raft_consensus.cc:491] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:43.314576 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a8409abe745918b8f11dc8590376a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:43.340806 11959 raft_consensus.cc:3054] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:43.341323 12718 raft_consensus.cc:1270] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20250113 08:59:43.314677 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487" is_pre_election: true
I20250113 08:59:43.342329 11887 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 8.
I20250113 08:59:43.315026 11859 leader_election.cc:304] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:43.291446 15440 raft_consensus.cc:2798] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:43.285979 15422 raft_consensus.cc:2743] T 364888f946914390a5a07be6735a4cc9 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:43.293512 14879 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Advancing to term 9
W20250113 08:59:43.366070 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 95323.4us
I20250113 08:59:43.314425 15251 consensus_queue.cc:1046] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20250113 08:59:43.368265 15251 raft_consensus.cc:3049] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:43.368613 15251 raft_consensus.cc:738] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:43.321926 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.369464 15251 consensus_queue.cc:260] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.370342 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a457bc5206c46d7b198dbff2b1c137c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.371536 12039 raft_consensus.cc:2405] T 9a457bc5206c46d7b198dbff2b1c137c P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20250113 08:59:43.371990 15228 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.372404 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc6c97986444348b1837c27c41338a" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:43.372834 14664 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.372505 15228 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.330166 15401 raft_consensus.cc:491] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.332553 12040 rpcz_store.cc:276] 0113 08:59:41.402508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.795931 (+393423us) service_pool.cc:225] Handling call
0113 08:59:43.085361 (+1289430us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:43.122346 (+ 36985us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.331309 (+208963us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x7b54003bcc54. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:43.331980 (+ 671us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54003bcc50. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.332096 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3047887360}
I20250113 08:59:43.383756 15401 raft_consensus.cc:513] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.384114 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1c553832b94166a0a6089104a74f6c" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.385187 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0"
I20250113 08:59:43.307958 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0640fd1a5740f1ac76ebcae33dfa25" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:43.385660 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:43.357081 15372 log.cc:926] Time spent T 1527f41e7e954abc8622a5f5da292602 P 37bd4bed3ee6423c8dac0cef3bf6882a: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250113 08:59:43.285233 14956 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:43.254528 15438 raft_consensus.cc:491] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.305012 15041 raft_consensus.cc:2798] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:59:43.347923 12720 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9.
W20250113 08:59:43.347986 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 915128us
I20250113 08:59:43.396384 11934 leader_election.cc:304] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:43.225184 15129 leader_election.cc:290] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.327323 15007 raft_consensus.cc:738] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:43.280635 15019 raft_consensus.cc:513] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.285734 15061 raft_consensus.cc:513] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.349746 15391 raft_consensus.cc:513] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.350211 15199 consensus_queue.cc:1046] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250113 08:59:43.351202 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.279987 15054 consensus_queue.cc:1046] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20250113 08:59:43.341799 12718 pending_rounds.cc:77] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting all ops after (but not including) 5
I20250113 08:59:43.426298 12718 pending_rounds.cc:99] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.6
I20250113 08:59:43.426113 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a2e0ccdce34284b13fa0924326b262" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.426628 12718 raft_consensus.cc:2911] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Aborting config change with OpId 6.6: Aborted: Op aborted by new leader
I20250113 08:59:43.352195 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364888f946914390a5a07be6735a4cc9" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.285919 15031 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.353569 15053 raft_consensus.cc:513] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.354128 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.354979 15146 consensus_queue.cc:1035] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250113 08:59:43.356299 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.438879 12721 raft_consensus.cc:2405] T 87a2e0ccdce34284b13fa0924326b262 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250113 08:59:43.357360 15413 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:43.361492 15344 raft_consensus.cc:491] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.395095 14956 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.370636 15251 raft_consensus.cc:3054] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:43.291653 14966 raft_consensus.cc:2798] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:43.422854 15189 raft_consensus.cc:491] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:43.374877 11932 leader_election.cc:304] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.375048 11959 raft_consensus.cc:2463] T 3e1a8409abe745918b8f11dc8590376a P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9c33777f6845c2848a2a6102a1c2ff in term 9.
I20250113 08:59:43.380573 15107 consensus_queue.cc:1035] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250113 08:59:43.381659 12686 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:43.472688 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.381742 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.385900 15401 leader_election.cc:290] T 8f63a7b00ea546f2865a4b5d730e35de P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.385928 12040 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 8: Already voted for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in this term.
I20250113 08:59:43.311803 11965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099cf849e7414101a8741339c72fa73e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:43.388854 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663c56b937924e7abebd7025ca892c1e" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.481160 11959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aa17121f5b429b8a32299911c7ef76" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.391492 12684 leader_election.cc:304] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:43.392980 12724 raft_consensus.cc:2463] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5f9c33777f6845c2848a2a6102a1c2ff [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 10.
I20250113 08:59:43.483448 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18dede64eda24dff8bfe2f979a4115d5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:43.311987 15223 raft_consensus.cc:491] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:43.346690 15440 raft_consensus.cc:491] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:43.399590 11934 leader_election.cc:304] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
W20250113 08:59:43.488674 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 89198.6us
I20250113 08:59:43.403885 15389 consensus_queue.cc:1035] T 531b33203b294592b3a1275dc74ab4f3 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.071s
I20250113 08:59:43.489444 11934 leader_election.cc:304] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:43.312608 15049 consensus_queue.cc:1046] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20250113 08:59:43.401389 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9d15ff319f43f0a1dd5ae717393675" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.373332 14664 raft_consensus.cc:513] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.403589 15309 raft_consensus.cc:2798] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250113 08:59:43.409590 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 126067us
I20250113 08:59:43.392729 15437 raft_consensus.cc:491] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:43.395725 15422 raft_consensus.cc:491] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:43.427652 11963 raft_consensus.cc:2388] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f9c33777f6845c2848a2a6102a1c2ff in current term 8: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:43.395179 15438 raft_consensus.cc:513] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.438934 11889 raft_consensus.cc:1235] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Rejecting Update request from peer 5e8883f5f83a47f4b9303dd1da4f59c0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250113 08:59:43.427085 12718 raft_consensus.cc:2883] T fc84bb347ec2413e9188b7253b0812ab P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20250113 08:59:43.468866 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.408866 15041 raft_consensus.cc:695] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 LEADER]: Becoming Leader. State: Replica: 5f9c33777f6845c2848a2a6102a1c2ff, State: Running, Role: LEADER
I20250113 08:59:43.361944 15388 raft_consensus.cc:2798] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:43.428943 15199 consensus_queue.cc:1225] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5f9c33777f6845c2848a2a6102a1c2ff log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:43.328212 15150 consensus_queue.cc:237] T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.427855 15261 raft_consensus.cc:2743] T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250113 08:59:43.476114 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 128640us
I20250113 08:59:43.474505 15392 consensus_queue.cc:1046] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20250113 08:59:43.478281 12039 raft_consensus.cc:2463] T e9223bd4b75148bc8c6a0349dcc26b9b P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 9.
I20250113 08:59:43.484817 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.503966 11934 leader_election.cc:304] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:43.476706 15434 leader_election.cc:290] T 7104dce7b01a423493a37b6ed694da07 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 election: Requested vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.478154 14966 raft_consensus.cc:491] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:43.508143 11859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 162953us
I20250113 08:59:43.486105 15189 raft_consensus.cc:3054] T 49d70ffc378340e785c4f8dde909f9ba P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:43.512377 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209260c370164f719208bd1404eef071" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.480331 15341 raft_consensus.cc:2798] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:59:43.494411 14865 raft_consensus.cc:2743] T a2bbae0b61a948339e4c443b9237ad87 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:43.492635 15440 raft_consensus.cc:3054] T 588088c74b20488da56f01a0b351a918 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:43.612149 12008 leader_election.cc:304] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff; no voters:
I20250113 08:59:43.473940 15344 raft_consensus.cc:513] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.519119 11856 leader_election.cc:304] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.500631 15309 raft_consensus.cc:491] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
W20250113 08:59:43.581867 15414 log.cc:926] Time spent T 4fe494493d33439a9e129bc5d8ab6654 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.753s user 0.003s sys 0.000s
I20250113 08:59:43.582098 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637bd1e4126142b481c15befc679b5ac" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.577112 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badac2289c8b4a10915cca20182fc423" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:43.577906 12722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38520 (request call id 847) took 1884 ms. Trace:
W20250113 08:59:43.577876 15383 log.cc:926] Time spent T 54d2a9b9e9134754acb8568b23bba884 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 1.348s user 0.000s sys 0.004s
I20250113 08:59:43.583604 14956 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.585233 15053 leader_election.cc:290] T 1527f41e7e954abc8622a5f5da292602 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.586239 15391 leader_election.cc:290] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:43.587672 15061 leader_election.cc:290] T 7d7522e543294e28ac66b2faf6001cbe P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:43.590260 15425 log.cc:926] Time spent T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.586s user 0.004s sys 0.007s
I20250113 08:59:43.479950 15007 consensus_queue.cc:260] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.597487 12726 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:43.576987 12035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1828) took 2269 ms. Trace:
I20250113 08:59:43.599848 12724 raft_consensus.cc:2371] T 043006a99fa54df3a6ee86ebd18003a7 P 5f9c33777f6845c2848a2a6102a1c2ff [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9. Re-sending same reply.
W20250113 08:59:43.633590 15446 log.cc:926] Time spent T 2c3345ca9b1640a4b4e6d2efd2ee061d P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20250113 08:59:43.603170 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
W20250113 08:59:43.604513 11932 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.129:33539, user_credentials={real_user=slave}} blocked reactor thread for 82357.9us
I20250113 08:59:43.617983 15309 raft_consensus.cc:3054] T a43de3c8f06b48358bcee58f9567bb3b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:43.470646 15413 raft_consensus.cc:513] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.518949 12040 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250113 08:59:43.676272 12040 pending_rounds.cc:77] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting all ops after (but not including) 6
I20250113 08:59:43.666498 12718 raft_consensus.cc:2371] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7. Re-sending same reply.
I20250113 08:59:43.676570 12040 pending_rounds.cc:99] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250113 08:59:43.676941 12040 raft_consensus.cc:2883] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250113 08:59:43.615242 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 111395us
W20250113 08:59:43.616381 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 97351.9us
I20250113 08:59:43.510195 15388 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:43.617612 11960 raft_consensus.cc:2463] T 637bd1e4126142b481c15befc679b5ac P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 9.
I20250113 08:59:43.572247 15019 leader_election.cc:290] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:43.598268 15224 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:43.636045 12722 rpcz_store.cc:276] 0113 08:59:41.693228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.467976 (+774748us) service_pool.cc:225] Handling call
0113 08:59:43.162151 (+694175us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:43.162825 (+ 674us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.577692 (+414867us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b54005ae450. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336608a09 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.577845 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1567105920}
I20250113 08:59:43.611908 11780 maintenance_manager.cc:418] P 48f8007a91af4c0c837ee737c6326d99: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033396
I20250113 08:59:43.522635 12034 raft_consensus.cc:2463] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:43.493772 15223 raft_consensus.cc:513] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.610364 14966 raft_consensus.cc:3054] T 79302a9220a149cb81cb2573664290a1 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:43.593763 12717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38504 (request call id 930) took 2026 ms (client timeout 2179 ms). Trace:
W20250113 08:59:43.696441 12717 rpcz_store.cc:269] 0113 08:59:41.567773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.852307 (+284534us) service_pool.cc:225] Handling call
0113 08:59:43.593528 (+1741221us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54007a0654. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:43.593689 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":849151360}
I20250113 08:59:43.696468 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043006a99fa54df3a6ee86ebd18003a7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a"
I20250113 08:59:43.697860 11960 raft_consensus.cc:3054] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:43.698942 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54973520c9884989a12742d4eb45994f" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.652129 11931 leader_election.cc:304] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
I20250113 08:59:43.508522 14879 raft_consensus.cc:513] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.600850 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3345ca9b1640a4b4e6d2efd2ee061d" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:43.664834 11858 leader_election.cc:336] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863): Timed out: RequestConsensusVote RPC to 127.10.115.130:41863 timed out after 1.734s (SENT)
I20250113 08:59:43.701805 11858 leader_election.cc:304] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:43.662798 12035 rpcz_store.cc:276] 0113 08:59:41.307239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.526470 (+219231us) service_pool.cc:225] Handling call
0113 08:59:41.526813 (+ 343us) raft_consensus.cc:1466] Updating replica for 0 ops
0113 08:59:41.527071 (+ 258us) raft_consensus.cc:1529] Early marking committed up to index 4
0113 08:59:42.865062 (+1337991us) log.cc:844] Serialized 14 byte log entry
0113 08:59:43.576390 (+711328us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b180028ebe0. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33c34a931 00007fc33c357831 00007fc33629408c 00007fc3336b34ec 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336cdbf5 00007fc3366013d0 00007fc336613666 00007fc3366241d7
0113 08:59:43.576542 (+ 152us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0113 08:59:43.576602 (+ 60us) raft_consensus.cc:1640] Marking committed up to 4
0113 08:59:43.576657 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.576743 (+ 86us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.576919 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1329405,"prepare.queue_time_us":1346294,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":3511,"raft.queue_time_us":131602,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":18696,"spinlock_wait_cycles":10359168,"thread_start_us":191745,"threads_started":2,"wal-append.queue_time_us":124372}
I20250113 08:59:43.703776 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7522e543294e28ac66b2faf6001cbe" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.605791 12720 raft_consensus.cc:2463] T 209260c370164f719208bd1404eef071 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 8.
I20250113 08:59:43.671617 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.632519 15389 raft_consensus.cc:2798] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election won for term 8
W20250113 08:59:43.675593 11787 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60016 (request call id 362) took 20145 ms (client timeout 15000 ms). Trace:
I20250113 08:59:43.705204 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39676b99e4434f11b6017bb867fb7471" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.679141 11934 leader_election.cc:304] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:43.680383 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6bba22615448a8acbbb898fb381116" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:43.707593 12724 raft_consensus.cc:2388] T 9a6bba22615448a8acbbb898fb381116 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 7: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:43.697803 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ace8d6d3d488583610962dabdee94" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.632136 15341 raft_consensus.cc:695] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:43.573048 15041 consensus_queue.cc:237] T 7104dce7b01a423493a37b6ed694da07 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.632563 15449 raft_consensus.cc:2798] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:43.659442 14736 raft_consensus.cc:491] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.744496 11959 raft_consensus.cc:2405] T 08aa17121f5b429b8a32299911c7ef76 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250113 08:59:43.663079 15228 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:43.709318 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105307f831744ff99c190ff6dd0db1a" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:43.750454 12718 raft_consensus.cc:2371] T a105307f831744ff99c190ff6dd0db1a P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7. Re-sending same reply.
I20250113 08:59:43.511090 15437 raft_consensus.cc:513] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.681666 15247 raft_consensus.cc:2798] T 69aa379773c047589ae2362850997a47 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:43.646723 15436 consensus_queue.cc:1046] T 80cc6c97986444348b1837c27c41338a P 5e8883f5f83a47f4b9303dd1da4f59c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20250113 08:59:43.699887 12040 raft_consensus.cc:2463] T 54973520c9884989a12742d4eb45994f P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in term 7.
I20250113 08:59:43.676620 15302 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.707095 12035 raft_consensus.cc:2463] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 5.
I20250113 08:59:43.733625 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de29a6701ddb4906a7eff4d34bf97ab5" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" is_pre_election: true
I20250113 08:59:43.763180 12722 raft_consensus.cc:2463] T de29a6701ddb4906a7eff4d34bf97ab5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
I20250113 08:59:43.643805 15401 consensus_queue.cc:1035] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20250113 08:59:43.665957 15439 raft_consensus.cc:2798] T badac2289c8b4a10915cca20182fc423 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:43.515101 15422 raft_consensus.cc:513] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.740669 15389 raft_consensus.cc:695] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:43.741665 11966 raft_consensus.cc:1212] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-7.5] Dedup: 7.5->[]
I20250113 08:59:43.682176 15388 raft_consensus.cc:3054] T fcc31099ea8040b7bfbfe7d779148ca5 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:43.687649 15007 raft_consensus.cc:3054] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 5f9c33777f6845c2848a2a6102a1c2ff [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:59:43.748374 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 83577.9us
W20250113 08:59:43.732067 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 80021.7us
I20250113 08:59:43.677913 15061 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20250113 08:59:43.758450 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a6087e1ad14cca8bb3f6d3e0bb1858" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.678913 14865 consensus_queue.cc:1035] T 79302a9220a149cb81cb2573664290a1 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.057s
I20250113 08:59:43.763931 12007 leader_election.cc:304] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:43.710307 15031 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:43.685470 15059 consensus_queue.cc:1035] T 272c58aa3da64d86aa8a3e3890e3a249 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20250113 08:59:43.742666 15270 raft_consensus.cc:2798] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250113 08:59:43.737077 15432 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.766396 11960 raft_consensus.cc:2463] T 043006a99fa54df3a6ee86ebd18003a7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 9.
I20250113 08:59:43.770045 12042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33878 (request call id 1826) took 2486 ms. Trace:
I20250113 08:59:43.739346 15019 raft_consensus.cc:491] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250113 08:59:43.747457 15443 log.cc:926] Time spent T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20250113 08:59:43.771905 15401 consensus_queue.cc:1225] T 3e1a8409abe745918b8f11dc8590376a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Peer 5f9c33777f6845c2848a2a6102a1c2ff log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:43.783761 15454 raft_consensus.cc:491] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.787243 15402 raft_consensus.cc:3049] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:59:43.739298 15449 raft_consensus.cc:491] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:43.737993 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23cb6607d3a463293b397049ef2d690" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.691936 15224 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.771886 15302 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
W20250113 08:59:43.839113 11839 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 37bd4bed3ee6423c8dac0cef3bf6882a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } }
I20250113 08:59:43.784333 15283 raft_consensus.cc:2798] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:59:43.754473 15104 raft_consensus.cc:2743] T 08aa17121f5b429b8a32299911c7ef76 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:43.778673 11856 leader_election.cc:304] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.846233 11966 raft_consensus.cc:1235] T 0afcc48db7a94f1bbc8242282f9da2dd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Rejecting Update request from peer 6ea00811b7d44b49b872ac5c03b91487 for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20250113 08:59:43.938565 15451 raft_consensus.cc:2798] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:59:43.939049 15451 raft_consensus.cc:491] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5f9c33777f6845c2848a2a6102a1c2ff)
I20250113 08:59:43.939392 15451 raft_consensus.cc:3054] T 39676b99e4434f11b6017bb867fb7471 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:59:43.939477 15179 raft_consensus.cc:2743] T e2677547aaee4bba86cf339fe13278bf P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:43.941681 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6ea00811b7d44b49b872ac5c03b91487"
I20250113 08:59:43.942303 11889 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52586 (request call id 2042) took 1342 ms. Trace:
I20250113 08:59:43.942515 11887 raft_consensus.cc:2388] T 7104dce7b01a423493a37b6ed694da07 P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in current term 8: Already voted for candidate 5f9c33777f6845c2848a2a6102a1c2ff in this term.
I20250113 08:59:43.942672 11889 rpcz_store.cc:276] 0113 08:59:42.599788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:43.233187 (+633399us) service_pool.cc:225] Handling call
0113 08:59:43.286874 (+ 53687us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:43.502942 (+216068us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.942057 (+439115us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b540038f254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:43.942243 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":348815616}
I20250113 08:59:43.873924 11963 raft_consensus.cc:2388] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ea00811b7d44b49b872ac5c03b91487 in current term 6: Already voted for candidate 37bd4bed3ee6423c8dac0cef3bf6882a in this term.
I20250113 08:59:43.944155 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2677547aaee4bba86cf339fe13278bf" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:43.937157 11856 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 158589us
I20250113 08:59:43.946906 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300f59ff3964ee28bb6ef0c1141395e" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.947789 12037 raft_consensus.cc:2405] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20250113 08:59:43.948565 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe494493d33439a9e129bc5d8ab6654" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.757795 14736 raft_consensus.cc:513] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:43.878487 11931 leader_election.cc:304] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters:
W20250113 08:59:43.950151 11931 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 71753.6us
I20250113 08:59:43.951298 15059 raft_consensus.cc:2798] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:59:43.951810 15059 raft_consensus.cc:491] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:43.952190 15059 raft_consensus.cc:3054] T 54973520c9884989a12742d4eb45994f P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:59:43.953532 12041 raft_consensus.cc:1235] T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 LEADER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:43.955246 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a99b7862446be849d4c843d8084ee" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
W20250113 08:59:43.877023 15447 log.cc:926] Time spent T 992519c5663649019a1ff9753075ddc7 P 6ea00811b7d44b49b872ac5c03b91487: Append to log took a long time: real 0.361s user 0.000s sys 0.001s
I20250113 08:59:43.882385 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992519c5663649019a1ff9753075ddc7" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.880890 12042 rpcz_store.cc:276] 0113 08:59:41.283169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.519129 (+235960us) service_pool.cc:225] Handling call
0113 08:59:41.555437 (+ 36308us) raft_consensus.cc:1466] Updating replica for 2 ops
0113 08:59:41.582063 (+ 26626us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:41.582553 (+ 490us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0113 08:59:41.630976 (+ 48423us) log.cc:844] Serialized 52 byte log entry
0113 08:59:41.846919 (+215943us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:42.915433 (+1068514us) log.cc:844] Serialized 14 byte log entry
0113 08:59:43.069020 (+153587us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.069138 (+ 118us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:43.491014 (+421876us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3362874ae 00007fc3362867f6 00007fc336286418 00007fc33628fdf1 00007fc33628fda9 00007fc33628fd39
0113 08:59:43.769664 (+278650us) raft_consensus.cc:1680] finished
0113 08:59:43.769792 (+ 128us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:43.769967 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1064840,"prepare.queue_time_us":1073319,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":158419,"raft.run_cpu_time_us":476,"raft.run_wall_time_us":31112,"spinlock_wait_cycles":10346880,"thread_start_us":292066,"threads_started":2,"wal-append.queue_time_us":534364}
I20250113 08:59:43.895242 15341 consensus_queue.cc:237] T 099cf849e7414101a8741339c72fa73e P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:43.964347 11890 raft_consensus.cc:1270] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:43.874369 14664 leader_election.cc:290] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
I20250113 08:59:43.895887 12039 raft_consensus.cc:2463] T a23cb6607d3a463293b397049ef2d690 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:43.979265 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:43.881549 15270 raft_consensus.cc:491] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250113 08:59:43.923676 12008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 264425us
I20250113 08:59:43.951359 14736 leader_election.cc:290] T cee55da481c64a6d885aa3ab442761cd P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231)
W20250113 08:59:43.964613 11934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.132:36511, user_credentials={real_user=slave}} blocked reactor thread for 285557us
I20250113 08:59:43.953079 11968 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43888 (request call id 2282) took 8718 ms. Trace:
I20250113 08:59:43.993171 11968 rpcz_store.cc:276] 0113 08:59:35.234335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:41.844222 (+6609887us) service_pool.cc:225] Handling call
0113 08:59:41.879932 (+ 35710us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.983152 (+1103220us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:43.952817 (+969665us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400380254. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc336609f51 00007fc33660899e 00007fc33b5672d9 00007fc334c33694 00007fc334c33622 00007fc334c33551 00007fc334c334cc 00007fc334c32782 00007fc334a34dfc 00007fc334a34236
0113 08:59:43.953016 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2535326464}
I20250113 08:59:43.945204 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:43.994765 11963 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8883f5f83a47f4b9303dd1da4f59c0 in term 6.
I20250113 08:59:43.971455 12037 raft_consensus.cc:2405] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37bd4bed3ee6423c8dac0cef3bf6882a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250113 08:59:43.895215 15402 raft_consensus.cc:738] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6ea00811b7d44b49b872ac5c03b91487, State: Running, Role: LEADER
W20250113 08:59:43.875725 15303 consensus_meta.cc:287] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0: Time spent flushing consensus metadata: real 0.525s user 0.010s sys 0.004s
I20250113 08:59:43.889581 15344 leader_election.cc:290] T 705a99b7862446be849d4c843d8084ee P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5e8883f5f83a47f4b9303dd1da4f59c0 (127.10.115.131:43231), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:43.706588 11787 rpcz_store.cc:269] 0113 08:59:23.590771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:23.723400 (+132629us) service_pool.cc:225] Handling call
0113 08:59:43.671089 (+19947689us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:59:28.655358 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0113 08:59:28.656220 (+ 862us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:59:28.656284 (+ 64us) write_op.cc:435] Acquired schema lock
0113 08:59:28.656306 (+ 22us) tablet.cc:659] Decoding operations
0113 08:59:28.658030 (+ 1724us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:59:28.658120 (+ 90us) write_op.cc:641] Partition lock acquired for write op
0113 08:59:28.658156 (+ 36us) tablet.cc:682] Acquiring locks for 15 operations
0113 08:59:28.659649 (+ 1493us) tablet.cc:698] Row locks acquired
0113 08:59:28.659677 (+ 28us) write_op.cc:260] PREPARE: finished
0113 08:59:28.659954 (+ 277us) write_op.cc:270] Start()
0113 08:59:28.660114 (+ 160us) write_op.cc:276] Timestamp: P: 1736758768659810 usec, L: 0
0113 08:59:28.660137 (+ 23us) op_driver.cc:348] REPLICATION: starting
0113 08:59:28.798877 (+138740us) log.cc:844] Serialized 6020 byte log entry
0113 08:59:31.227019 (+2428142us) op_driver.cc:464] REPLICATION: finished
0113 08:59:35.254638 (+4027619us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b440002494c. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc3335142d1 00007fc3335e2055 00007fc3335e2a72 00007fc336297da1 00007fc33629368a 00007fc3336b310e 00007fc33c2c47f9 00007fc3336d035b 00007fc3336ce75f 00007fc3336d08c2 00007fc33a01cd20
0113 08:59:35.906494 (+651856us) write_op.cc:301] APPLY: starting
0113 08:59:35.906635 (+ 141us) tablet.cc:1364] starting BulkCheckPresence
0113 08:59:35.907100 (+ 465us) tablet.cc:1367] finished BulkCheckPresence
0113 08:59:35.907132 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:59:36.038687 (+131555us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:59:36.039169 (+ 482us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0113 08:59:36.039240 (+ 71us) write_op.cc:312] APPLY: finished
0113 08:59:36.041349 (+ 2109us) log.cc:844] Serialized 107 byte log entry
0113 08:59:37.319053 (+1277704us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:37.319865 (+ 812us) write_op.cc:454] Released schema lock
0113 08:59:37.320032 (+ 167us) write_op.cc:341] FINISH: Updating metrics
0113 08:59:37.320963 (+ 931us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:59:37.320998 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1810735,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":4614069,"apply.run_cpu_time_us":88199,"apply.run_wall_time_us":1414367,"num_ops":15,"prepare.queue_time_us":413929,"prepare.run_cpu_time_us":10731,"prepare.run_wall_time_us":334173,"raft.queue_time_us":357150,"raft.run_cpu_time_us":74367,"raft.run_wall_time_us":4939540,"replication_time_us":2517683,"spinlock_wait_cycles":25511808,"thread_start_us":6308730,"threads_started":5,"wal-append.queue_time_us":1645029,"wal-append.run_cpu_time_us":81578,"wal-append.run_wall_time_us":3844986}]]}
I20250113 08:59:43.985400 11856 leader_election.cc:304] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:43.987102 12041 raft_consensus.cc:1235] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Rejecting Update request from peer 37bd4bed3ee6423c8dac0cef3bf6882a for earlier term 8. Current term is 9. Ops: []
I20250113 08:59:43.880636 15438 leader_election.cc:290] T 272c58aa3da64d86aa8a3e3890e3a249 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:43.900262 15449 raft_consensus.cc:3054] T 4a9d15ff319f43f0a1dd5ae717393675 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:59:44.002094 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:44.003774 11859 leader_election.cc:336] T ce0640fd1a5740f1ac76ebcae33dfa25 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511): Timed out: RequestConsensusVote RPC to 127.10.115.132:36511 timed out after 1.586s (SENT)
I20250113 08:59:44.003962 11934 leader_election.cc:304] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5f9c33777f6845c2848a2a6102a1c2ff; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:44.011878 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee55da481c64a6d885aa3ab442761cd" candidate_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" is_pre_election: true
I20250113 08:59:44.014914 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79302a9220a149cb81cb2573664290a1" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.016320 11963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9223bd4b75148bc8c6a0349dcc26b9b" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.018253 15413 leader_election.cc:290] T 39676b99e4434f11b6017bb867fb7471 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:44.018976 15455 log.cc:926] Time spent T 209260c370164f719208bd1404eef071 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
W20250113 08:59:44.025791 15452 log.cc:926] Time spent T 89d1890dc82d4bd889d0300c9fed050d P 5f9c33777f6845c2848a2a6102a1c2ff: Append to log took a long time: real 0.283s user 0.001s sys 0.000s
I20250113 08:59:43.892714 15432 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:44.030139 11934 leader_election.cc:304] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5f9c33777f6845c2848a2a6102a1c2ff, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:43.999197 15270 raft_consensus.cc:3054] T e9223bd4b75148bc8c6a0349dcc26b9b P 37bd4bed3ee6423c8dac0cef3bf6882a [term 9 FOLLOWER]: Advancing to term 10
I20250113 08:59:43.981068 15283 raft_consensus.cc:695] T 364888f946914390a5a07be6735a4cc9 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 LEADER]: Becoming Leader. State: Replica: 37bd4bed3ee6423c8dac0cef3bf6882a, State: Running, Role: LEADER
I20250113 08:59:43.988137 15031 raft_consensus.cc:491] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:43.890025 15019 raft_consensus.cc:513] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:44.036156 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2a9b9e9134754acb8568b23bba884" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.038519 11934 leader_election.cc:304] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a; no voters: 5e8883f5f83a47f4b9303dd1da4f59c0, 5f9c33777f6845c2848a2a6102a1c2ff
I20250113 08:59:44.011480 15303 raft_consensus.cc:2743] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:44.057905 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8210eb7687124bcfbc64e314e532799e" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.060094 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1527f41e7e954abc8622a5f5da292602" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
W20250113 08:59:44.060534 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 296668us
I20250113 08:59:43.898661 15454 raft_consensus.cc:513] T 89d1890dc82d4bd889d0300c9fed050d P 6ea00811b7d44b49b872ac5c03b91487 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:44.063181 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afcc48db7a94f1bbc8242282f9da2dd" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.041184 15177 raft_consensus.cc:2743] T 49d70ffc378340e785c4f8dde909f9ba P 37bd4bed3ee6423c8dac0cef3bf6882a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:44.031968 15463 raft_consensus.cc:2798] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250113 08:59:44.064031 11858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 198826us
I20250113 08:59:44.081257 12038 raft_consensus.cc:1235] T 18dede64eda24dff8bfe2f979a4115d5 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 8 FOLLOWER]: Rejecting Update request from peer 5f9c33777f6845c2848a2a6102a1c2ff for earlier term 7. Current term is 8. Ops: []
I20250113 08:59:44.064117 12037 raft_consensus.cc:1270] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Refusing update from remote peer 6ea00811b7d44b49b872ac5c03b91487: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250113 08:59:44.043257 15394 raft_consensus.cc:2743] T 70a6087e1ad14cca8bb3f6d3e0bb1858 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 08:59:44.041164 14879 leader_election.cc:290] T 05d6fc5c56554478ae416a917a542304 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 election: Requested vote from peers 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
I20250113 08:59:44.063259 12007 leader_election.cc:304] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:44.094098 11858 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:44.098918 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f63a7b00ea546f2865a4b5d730e35de" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.100579 11858 leader_election.cc:304] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487; no voters:
I20250113 08:59:44.101543 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272c58aa3da64d86aa8a3e3890e3a249" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.102890 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588088c74b20488da56f01a0b351a918" candidate_uuid: "6ea00811b7d44b49b872ac5c03b91487" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.065043 11960 raft_consensus.cc:2463] T 992519c5663649019a1ff9753075ddc7 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 7.
W20250113 08:59:44.108166 12007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 44977.1us
I20250113 08:59:44.103574 11968 raft_consensus.cc:2463] T 588088c74b20488da56f01a0b351a918 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea00811b7d44b49b872ac5c03b91487 in term 6.
I20250113 08:59:44.112934 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" is_pre_election: true
I20250113 08:59:44.062965 15437 leader_election.cc:290] T 4fe494493d33439a9e129bc5d8ab6654 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:44.117784 11858 leader_election.cc:304] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:44.061420 15144 raft_consensus.cc:491] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:44.124521 11858 leader_election.cc:304] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ea00811b7d44b49b872ac5c03b91487; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 5e8883f5f83a47f4b9303dd1da4f59c0
I20250113 08:59:44.127411 11968 raft_consensus.cc:1270] T b6a40dc5e61345558c943df4be928468 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Refusing update from remote peer 5f9c33777f6845c2848a2a6102a1c2ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250113 08:59:44.092916 15463 raft_consensus.cc:491] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:59:44.070248 15061 raft_consensus.cc:2743] T 663c56b937924e7abebd7025ca892c1e P 37bd4bed3ee6423c8dac0cef3bf6882a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:44.051528 15344 consensus_queue.cc:1035] T 8f63a7b00ea546f2865a4b5d730e35de P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20250113 08:59:44.112690 14790 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:44.008193 15389 consensus_queue.cc:237] T 7dda9aa8769e4146a4dabeadd588c734 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:44.114462 15146 raft_consensus.cc:2743] T 0afcc48db7a94f1bbc8242282f9da2dd P 6ea00811b7d44b49b872ac5c03b91487 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250113 08:59:44.118249 15231 raft_consensus.cc:2798] T 043006a99fa54df3a6ee86ebd18003a7 P 6ea00811b7d44b49b872ac5c03b91487 [term 9 FOLLOWER]: Leader election won for term 9
I20250113 08:59:44.053411 15031 raft_consensus.cc:513] T fc84bb347ec2413e9188b7253b0812ab P 37bd4bed3ee6423c8dac0cef3bf6882a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } attrs { replace: true } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: NON_VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: true } }
I20250113 08:59:44.118167 15422 leader_election.cc:290] T b300f59ff3964ee28bb6ef0c1141395e P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
I20250113 08:59:44.142828 15463 raft_consensus.cc:3054] T a23cb6607d3a463293b397049ef2d690 P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Advancing to term 7
W20250113 08:59:44.171979 12723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38520 (request call id 859) took 1815 ms (client timeout 2216 ms). Trace:
I20250113 08:59:44.126893 15413 raft_consensus.cc:491] T faa96abb4c9940e4aa27327deeb68af0 P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6ea00811b7d44b49b872ac5c03b91487)
I20250113 08:59:44.106631 15401 consensus_queue.cc:1035] T fc84bb347ec2413e9188b7253b0812ab P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20250113 08:59:44.138350 15391 raft_consensus.cc:2743] T 8210eb7687124bcfbc64e314e532799e P 6ea00811b7d44b49b872ac5c03b91487 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:59:42.974537 11892 rpcz_store.cc:276] 0113 08:59:41.856496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.807008 (+950512us) service_pool.cc:225] Handling call
0113 08:59:42.815471 (+ 8463us) raft_consensus.cc:1466] Updating replica for 1 ops
0113 08:59:42.836005 (+ 20534us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:59:42.836051 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0113 08:59:42.836604 (+ 553us) log.cc:844] Serialized 26 byte log entry
0113 08:59:42.907080 (+ 70476us) raft_consensus.cc:1640] Marking committed up to 2
0113 08:59:42.919651 (+ 12571us) log.cc:844] Serialized 14 byte log entry
0113 08:59:42.933702 (+ 14051us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:59:42.933840 (+ 138us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:59:42.973878 (+ 40038us) raft_consensus.cc:1680] finished
0113 08:59:42.973897 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:59:42.974046 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11844,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52330,"raft.queue_time_us":22089,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":18779,"thread_start_us":37310,"threads_started":1,"wal-append.queue_time_us":121203}
I20250113 08:59:44.134475 15129 raft_consensus.cc:2743] T 54d2a9b9e9134754acb8568b23bba884 P 6ea00811b7d44b49b872ac5c03b91487 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250113 08:59:44.128968 15473 raft_consensus.cc:2798] T 637bd1e4126142b481c15befc679b5ac P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250113 08:59:44.114190 15199 consensus_queue.cc:1035] T a105307f831744ff99c190ff6dd0db1a P 6ea00811b7d44b49b872ac5c03b91487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20250113 08:59:44.134737 15434 raft_consensus.cc:491] T 8210eb7687124bcfbc64e314e532799e P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 37bd4bed3ee6423c8dac0cef3bf6882a)
I20250113 08:59:44.190080 15466 raft_consensus.cc:2798] T 9a6bba22615448a8acbbb898fb381116 P 37bd4bed3ee6423c8dac0cef3bf6882a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 08:59:44.222344 15393 raft_consensus.cc:3049] T 18dede64eda24dff8bfe2f979a4115d5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 LEADER]: Stepping down as leader of term 7
I20250113 08:59:44.198386 15303 raft_consensus.cc:491] T a105307f831744ff99c190ff6dd0db1a P 5e8883f5f83a47f4b9303dd1da4f59c0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:59:44.203797 11964 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43872 (request call id 2306) took 9353 ms. Trace:
I20250113 08:59:44.204042 15302 leader_election.cc:290] T fc84bb347ec2413e9188b7253b0812ab P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 37bd4bed3ee6423c8dac0cef3bf6882a (127.10.115.130:41863)
W20250113 08:59:44.211962 12686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.130:41863, user_credentials={real_user=slave}} blocked reactor thread for 830350us
I20250113 08:59:44.212347 15418 leader_election.cc:290] T ce0640fd1a5740f1ac76ebcae33dfa25 P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6ea00811b7d44b49b872ac5c03b91487 (127.10.115.129:33539), 5f9c33777f6845c2848a2a6102a1c2ff (127.10.115.132:36511)
W20250113 08:59:44.212688 15461 log.cc:926] Time spent T 6a1ace8d6d3d488583610962dabdee94 P 5e8883f5f83a47f4b9303dd1da4f59c0: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20250113 08:59:44.064859 15402 consensus_queue.cc:260] T a2bbae0b61a948339e4c443b9237ad87 P 6ea00811b7d44b49b872ac5c03b91487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5f9c33777f6845c2848a2a6102a1c2ff" member_type: VOTER last_known_addr { host: "127.10.115.132" port: 36511 } attrs { promote: false } }
I20250113 08:59:44.213508 15214 raft_consensus.cc:2798] T 663c56b937924e7abebd7025ca892c1e P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250113 08:59:44.214059 15251 raft_consensus.cc:2743] T 9a457bc5206c46d7b198dbff2b1c137c P 5f9c33777f6845c2848a2a6102a1c2ff [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250113 08:59:44.139487 15144 raft_consensus.cc:513] T a47a39083de44711b040bcf422953893 P 6ea00811b7d44b49b872ac5c03b91487 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea00811b7d44b49b872ac5c03b91487" member_type: VOTER last_known_addr { host: "127.10.115.129" port: 33539 } } peers { permanent_uuid: "37bd4bed3ee6423c8dac0cef3bf6882a" member_type: VOTER last_known_addr { host: "127.10.115.130" port: 41863 } } peers { permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 } }
I20250113 08:59:44.216548 15049 raft_consensus.cc:491] T fcc31099ea8040b7bfbfe7d779148ca5 P 5f9c33777f6845c2848a2a6102a1c2ff [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5e8883f5f83a47f4b9303dd1da4f59c0)
I20250113 08:59:44.218047 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7104dce7b01a423493a37b6ed694da07" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:44.220294 12724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d6fc5c56554478ae416a917a542304" candidate_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5f9c33777f6845c2848a2a6102a1c2ff"
I20250113 08:59:44.434955 12724 raft_consensus.cc:3054] T 05d6fc5c56554478ae416a917a542304 P 5f9c33777f6845c2848a2a6102a1c2ff [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:59:44.090912 14865 consensus_queue.cc:1046] T 05d6fc5c56554478ae416a917a542304 P 37bd4bed3ee6423c8dac0cef3bf6882a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e8883f5f83a47f4b9303dd1da4f59c0" member_type: VOTER last_known_addr { host: "127.10.115.131" port: 43231 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
W20250113 08:59:44.241317 12684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.115.131:43231, user_credentials={real_user=slave}} blocked reactor thread for 849886us
W20250113 08:59:44.227093 12723 rpcz_store.cc:269] 0113 08:59:42.356250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:59:42.419477 (+ 63227us) service_pool.cc:225] Handling call
0113 08:59:44.171770 (+1752293us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b54006eebd4. stack: 00007fc33369e853 00007fc33369e750 00007fc332ff0584 00007fc33cae5ff4 00007fc33cae5f89 00007fc33658e6d1 00007fc33660d2ae 00007fc33b567c17 00007fc334c35c14 00007fc334c35ba2 00007fc334c35ad1 00007fc334c35a4c 00007fc334c34d02 00007fc334a34dfc 00007fc334a34236 00007fc334a33bdf
0113 08:59:44.171914 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30559104}
I20250113 08:59:44.217283 15392 consensus_queue.cc:1225] T 218ea9985e1d40a48892a8e01aaaef39 P 5f9c33777f6845c2848a2a6102a1c2ff [LEADER]: Peer 6ea00811b7d44b49b872ac5c03b91487 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250113 08:59:44.274914 12007 leader_election.cc:304] T 0afcc48db7a94f1bbc8242282f9da2dd P 5e8883f5f83a47f4b9303dd1da4f59c0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e8883f5f83a47f4b9303dd1da4f59c0; no voters: 37bd4bed3ee6423c8dac0cef3bf6882a, 6ea00811b7d44b49b872ac5c03b91487
I20250113 08:59:44.281859 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "